[trustable-software] Don't pretend that Reliability means Safety (was Re: Additional requirement for trustability)

Sally Penni sjpenni at yahoo.co.uk
Wed Sep 12 11:01:58 BST 2018


Interesting 

Best Wishes 
 
Sally 
Miss Sally S-J Penni, (Fellow of RSA and CCMI) 
Barrister at Law, Vice Chair of AWB, Trustee and NED 
Founder/CEO  of Womeninthelawuk.co.uk
LinkedIn 
Twitter handle  @sallypenni1      @womeninthelawuk

Northern Power Woman, Power List 2017,
Winner Diversity Champion Award, awarded by UK Diversity Legal Awards 2016,
Winner of WE ARE THE CITY Rising Star Awards sponsored by The Times 2017,
Winner of HSBC & Forward Ladies Women in Business Rising Star Award 2017



> On 12 Sep 2018, at 09:07, Paul Sherwood <paul.sherwood at codethink.co.uk> wrote:
> 
> Hi Jonathan,
> thanks for this. Please see my comments inline.
> 
>> On 2018-09-11 15:23, Jonathan Moore wrote:
>> In automotive at least the department is generally called
>> Homologation. These people are charged with a couple of key
>> responsibilities:
>> 1. Assembling the pieces of paper for a product action into the
>> archive that will get released in discovery and may help the argument
>> that the OEM adopted State of the Art, wasn't
>> incompetent/lazy/ignorant etc.
> 
> Yup, understood. "Let's huddle together and make sure we're no worse than the herd."
> 
>> 2. When a new product action is announced identifying the pieces of
>> paper that engineering need to create so they can estimate the effort,
>> plan and deliver them prior to launch 18-24 months later.
> 
> This has appeared to 'work' for a very long time, so it's kind of understandable. However, based on what I already know about autonomous, I think all bets are off for grandfathering in proven-in-use 'safe' bits :)
> 
>> For a world car this can be quite complex and in part I think what's
>> driving the effort, in the US at least, to attempt to prevent
>> individual states from creating their own unique legislation
>> regulating automated driving vehicles. I'm not 100% sure the value of
>> this because algorithms of the future are likely going to have to deal
>> with handedness, road sign differences, local conventions anyway but
>> that's as I understand the approach at the moment.
> 
> Personally I don't have high confidence that individual states/countries will create uniformly appropriate and effective legislation. Do you?
> 
> <snip>
> 
>> Keep in mind most (all) automotive OEMS and their reputable Tier One's
>> are members of the SAE. This 'club' is intended as the repository of
>> all things best practice, state of the art and forms the back bone of
>> most liability defenses and is an attempt at a sort of herd immunity.
> 
> Amazingly, I hadn't even heard of this organisation :/
> 
>> The argument is something like, the last model of the vehicle was
>> demonstrably safe (didn't have this particular fault), here are the
>> changes we made (they were small and incremental improvements), here's
>> the statistics, data, studies, conclusions of the experts in how to
>> engineer x safely. Here's our evidence we followed those
>> recommendations and evidence that what we changed made improvements.
>> SAE have a very close relationship with ISO and IEC and OEMs like the
>> voluntary adoption of their standards because of the collective
>> immunity by adopting best practice.
> 
> Even if it were true that vehicle x+1 only has a manageable small set of changes vs vehicle x (which it often isn't these days afaict), a small change in any component can clearly have unexpected implications for other elements of the system, so I think from an engineering perspective the basis of the argument is false.
> 
>> In Europe it's not so clear to me that IEC 61508 and ISO 26262 are in
>> fact not legal requirements. The three key Type Approval regulations
>> (that apply since the Machinery Directive excludes on road vehicles) -
>> have language like "include but not limited to" when it comes to
>> identifying risk and safety based actions, or they specifically
>> mandate E/E systems by name necessary for safety which current best
>> practice/state of the art brings them into 508 and 262. Incidentally
>> there is a proposal for regulation in progress since May and I've not
>> met an European OEM homologation engineer that believes these
>> 'guidelines' are anything other than required i.e. unwilling to let
>> the company take a risk of finding out in a court. The directives in
>> force in Europe are 661/209, 78/2009 and 79/2009 and freely available
>> from EUR-Lex.  If the call is successful it looks like the plan is to
>> repeal these in favour of a new regulatory document.
> 
> Interesting. Any early signals about what approach that will take?
> 
> A couple of people have mentioned SOTIF to me recently, and IIUC that's basically the standards folks gently switching horses without admitting they were wrong before...
> 
>> Ultimately I think the legal compliance or not discussion is a dead
>> end. Toyota taught us the courts don’t even care to see a specific
>> bug if the overwhelming evidence is that the product wasn't the result
>> of reasonable engineering best practice.
> 
> Good point! :-)
> 
> And let me state here, unequivocally, so there's a bit of 'prior art' from this community irrespective of what the standards folks and other clubs/herds want to claim in court later:
> 
> ***
> 
> As researched and thoroughly documented by MIT and others, safety is a system-level property, not correlated with reliability of components.
> 
> I've recreated the assumptions table from Nancy Leveson's book at [1] and I quote:
> 
> "High reliability is neither necessary nor sufficient for safety."
> 
> "Highly reliable software is not necessarily safe. Increasing software reliability will have little impact on software safety."
> 
> Anyone in 2018 failing to consider safety at the system level, including all of the potential interactions between components, and the socio-technical interactions from outside that affect how we construct systems, is obviously **not** applying reasonable engineering best practice, since the research including implementation methods is freely available and the methods are commercially justifiable.
> 
> ***
> 
>> The fundamental question is one that has been debated and argued for
>> many decades now. Do you believe it is possible to predict the failure
>> of electronics using statistical techniques or not?
> 
> Wrong question. Reliability != Safety.
> 
>> If you don't and
>> there are standards e.g. 13849 that don't, or do and there are
>> standards that are firmly rooted in this 61508 the fact remains we
>> have to describe / defend / justify to our peers, families, friends
>> and children what we are doing and why it is safe.
> 
> True. Let's focus on that.
> 
>> If we get it wrong
>> there will be a chorus of 'told you so' and experts lining up to
>> defend the plaintiff and I don't know any automotive Homologation
>> Engineers whose organization will allow them to take a course of
>> action that puts them out of step with their peers and colleagues -
>> either within or external to their company. ie if we take a new
>> position on a model year 2020 vehicle and it's safer why did the
>> company not adopt this new approach on all their other carlines.
> 
> Irrespective of how well or badly people may think that the previous approaches have worked, all bets are off for autonomous, and anyone who suggests otherwise must be either lying, lazy, stupid or incompetent imo.
> 
>> I have some notes on this equation if folk are interested in the
>> derivation. What we are wrestling with is the RHS equation and I think
>> the conclusion that this is basically unknowable / too complex to
>> calculate from first principals (e.g. as 61508/26262) requires
> 
> Sorry, I must be missing something. Is this an equation for reliability, in which case it's irrelevant as stated above? Or something else...
> 
>> don’t have 1E9 hours of proven in use and in automotive at least the
> 
> "Proven in use" is not engineering IMO, it's just folklore.
> 
>> use case is so general purpose and since we don’t recover units from
>> the field at end of life (crushing vs. e.g. lifting a plane from the
>> ocean floor) and …
>> An alternative approach might be to select from several design
>> alternatives - the one that is least likely to fail and then combine
>> that with STAMP/Reliability prediction on the lower complexity
>> subsystems/subcompents/elements.
> 
> Possibly. Let's attempt to properly describe the problem/intents before we jump to potential solutions, though :)
> 
> br
> Paul
> 
> [1] https://gitlab.com/trustable/overview/wikis/safety/stpa-notes
> 
> _______________________________________________
> trustable-software mailing list
> trustable-software at lists.trustable.io
> https://lists.trustable.io/cgi-bin/mailman/listinfo/trustable-software
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.trustable.io/pipermail/trustable-software/attachments/20180912/54fabe5b/attachment-0001.html>


More information about the trustable-software mailing list