[trustable-software] Evidence Relationships in SDLC

[Paul Sherwood]

Hi Edmund

please see my comments inline...

TL; DR i think you've taken some liberties with your definitions :-)

On 2018-03-28 16:26, trustable at panic.fluff.org wrote:
>       I wonder if you might be able to address this problem for me.
> I'm sort of stuck. I've been using the 'trustable' discussion to help
> get to something which I can implement.
>  https://gitlab.com/trustable/workflow/blob/master/markdown/basics.md
> 
> we discuss these relationships
> 
> So I believe the following to be the case
> 
>  	* each t.test MUST be associated with one or more t.requirement
> 	* t.function is a unit of execution of t.software
> 	* each t.function MUST be associated with one or more t.requirement
> 	* each t.function MUST be validated by t.test
> 	* each t.requirement MUST be unitary/cohesive
> 	* each t.requirement MUST have one or more t.test

This is currently still at the hypothesis stage, with various thought 
experiments and incomplete empirical evidence.

> So let's agree what a requirement is
> 
>   https://en.wikipedia.org/wiki/Requirement

I'm surprised at you - we don't agree something just because it's 
claimed on wikipedia :)

> And such a requirement should be Unitary and Cohesive. That is to say
> that it should address one and only one behavior.

I'm afraid I no longer agree with that, hence there is no such assertion 
in the basics.md file you mentioned above.

Dan Firth on this list provided a logic argument which appeared to nail 
it last year:

https://lists.trustable.io/pipermail/trustable-software/2018-January/000284.html

> Thus if you need to
> validate more than one behavior you require more than one requirement.
> I'm going to use this definition for t.requirement

No, based on the above.

> Now let's define what a function is. That is to say a callable unit of
> executable instruction.
> 
>     https://en.wikipedia.org/wiki/Subroutine
> 
> I'm going to use this definition for t.function

There's already a definition of t.function in the document, by all means 
propose an improvement there, but again wikipedia may or may not be 
helpful.

> Now let's define what a test is.
> 
>    https://en.wikipedia.org/wiki/Test_method
> 
> That is to say a definitive procedure that determines one or more
> characteristics of a given process execution. I'm going to use this as 
> a
> definition of t.test

Same criticisms as above - we already have some first-pass logic 
describing t.tests.

Can you propose improvements to that, or refute it?

> So what relationships are we trying to record
> 
>  	* t.requirement has one or more t.test

Yes, ok

> 	* t.requirement has one or more t.function

No: 'has' is the wrong verb, for sure. 'provided by' perhaps.

> 	* t.function has one or more t.test

Again, not 'has'. Currently we have "Each t.function MUST be validated 
by t.tests"

> 	* t.function has one or more t.requirement

Maybe s/has/contributes to/

> 	* t.test has one or more t.function

s/has/MAY validate some or all of/

> 	* t.test has one or more t.requirement

s/has/MAY validate some or all of/

> 	* t.test may exercise one or more t.requirement which in turn 
> exercises
> one or more t.function

possibly :)

> So here is the problem.
> 
>     We two requirements t.requirement[a] and t.requirement[b]
> 
>     We have a t.function[1] which is used in the implementation of
> t.requirement[a] & t.requirement[b]
> 
>     We have a t.function[2] which used in the implementation of
> t.requirement[a] & t.requirement[b]
> 
>     We have a t.test[1] for t.function[1]
> 
>     We have a t.test[2] for t.function[2]
> 
>     We gain a new requirement t.requirement[c]
> 
>     We have a  t.function[3] which is used in the implementation of
> r.requirement[c]
> 
>     We have a t.test[3] for t.function[3]
> 
>     We now discover that t.test[1] fails with the addition of
> t.function[3]
> 
>     We now discover that the root cause of t.test[1] failure is the
> interaction between t.function[3] and t.function[1]

The above seems plausible, even for the t.* definitions in basics.md

> Problem Constraints
> 
>    At any moment, we must be able to validate the relationship between
> t.requirement, t.function, t.test and the t.change. Ideally this
> relationship should not depend on the developer populating the metadata
> most likely incorrectly. Ideally this meta data should reside in the
> SCM tool perhaps in things like Commit Messages or Notes (git notes).
> 
> Here is the question ?
> 
>    How do we manage the linkage between these things without loosing 
> the
> relationship between the t.function[1] and t.requirement[a] ?
> 
>    Why is this a problem ? During the implementation of 
> t.requirement[c]
> we have to refactor the t.function[1] and t.test[1] from
> t.requirement[a]

Strictly 'refactor' is not the word to use, but I take your point, and I 
think it's unavoidable.

All this boils down to is that introducing any new code can expose 
previously unseen problems in existing code.

It is well understood that changing a single line of code can totally 
change the behaviour of a whole system.

What I'm not sure about is whether you are considering some shortcut to 
resolve the exposed problem(s) in this scenario? If you are, I'd really 
like to understand it.

>          Really value your opinion on this
>           Edmund
> 
> 
> ps I believe I have a horrid HACK to do this but was looking for
> something more beautiful

Would be great if you could share the hack? Maybe others can beautify...

br
Paul