[C-safe-secure-studygroup] Static vs dynamic analysis: who cares?

Barton Miller bart at cs.wisc.edu
Fri Jan 6 17:10:57 UTC 2017


"Flaw" is certainly a reasonable synonym for weakness.  (weakness == 
flaw == bug, for most folks)

I think that the key is to /not /use vulnerability where we are talking 
about flaws in the code, unless it is associated with a specific exploit.

--bart


On 1/6/2017 11:04 AM, Robert Seacord wrote:
> On the subject(s) of vocabulary and free standards, it is preferable 
> to use standard ISO definitions in ISO documents.
>
> The ISO/IEC 2382:2015(en) Information technology — Vocabulary is 
> actually available for free from here:
>
> https://www.iso.org/obp/ui/#iso:std:iso-iec:2382:ed-1:v1:en
>
> If we can't find a definition we are looking for, we can add one.  TS 
> 17961 added the following definitions:
>
> https://gitlab.com/trustable/C_Safety_and_Security_Rules_Study_Group/wikis/terms-and-definitions
>
> Instead of "weaknesses", I think we used the following term:
>
>
>       Security flaw
>
> Defect that poses a potential security risk
>
> We can certainly revisit this definition, as it is lacking form a 
> safety perspective. We can also include definitions form normative 
> references, which we do here: 
> https://gitlab.com/trustable/C_Safety_and_Security_Rules_Study_Group/wikis/normative-references
>
> Another good source of definitions might be WG23: 
> http://www.open-std.org/JTC1/SC22/WG23/docs/documents
>
> There are also some very lovely documents up there such as
>
> TR 24772-3 C Specific Language Vulnerabilities with Concurrency, 
> submitted by David Keaton 
> <http://www.open-std.org/JTC1/SC22/WG23/docs/ISO-IECJTC1-SC22-WG23_N0682-tr24772-3-post-meeting-46-with-concurrency-edits.docx>
>
>
> Which may be a great source for C specific language vulnerabilities 
> with Concurrency (just a guess).
>
> If we would prefer to use terms defined outside of ISO/IEC documents, 
> I think it is probably best that we just add them to our document. In 
> fact, I just went ahead and added this to the wiki:
>
>
>       Safety flaw
>
> Defect that poses a potential safety risk
>
> It was fun and easy! Everyone should try this.  8^)  The benefit of 
> changing things in the wiki (or wherever, pending instructions from 
> Laurence and Chris Polin) is that if no one ever changes this it will 
> end up in the International Standard. Stuff posted to the list doesn't 
> end up anywhere except on the list.
>
> Thanks,
> rCs
>
> On Fri, Jan 6, 2017 at 11:26 AM, Barton Miller via 
> C-safe-secure-studygroup <c-safe-secure-studygroup at lists.trustable.io 
> <mailto:c-safe-secure-studygroup at lists.trustable.io>> wrote:
>
>     +1
>
>     Good to use the accepted terminology.  (Though I have to admit,
>     when speaking, I often use the wrong term!)
>
>     --bart
>
>
>     On 1/6/2017 10:15 AM, Joe Jarzombek via C-safe-secure-studygroup
>     wrote:
>>
>>     All –
>>
>>     We are actually addressing ‘weaknesses’; not just
>>     ‘vulnerabilities’ in the C language.  Unless there is a specific
>>     exploit identified that takes advantage of the flaw, it is only a
>>     weakness (which could be a quality or security weakness).  If an
>>     exploit is identified that takes advantage of the flaw, then that
>>     flaw is a vulnerability.
>>
>>     Alternative statement: /The existence (even if only theoretical)
>>     of an exploit designed to take advantage of a _weakness
>>     <https://capec.mitre.org/about/glossary.html>_ (or multiple
>>     weaknesses) and achieve a _negative technical impact
>>     <https://capec.mitre.org/about/glossary.html>_ is what makes a
>>     weakness a _vulnerability
>>     <https://capec.mitre.org/about/glossary.html>_./
>>
>>     Derived from definitions in ITU-T CYBEX 1500 series and the MITRE
>>     websites for CWE, CVE and CAPEC:
>>
>>     *Weakness: *mistake or flaw condition in architecture, design,
>>     code, or process that, if left unaddressed, could under the
>>     proper conditions contribute to a _cyber-enabled capability
>>     <https://capec.mitre.org/about/glossary.html>_ being vulnerable
>>     to exploitation; represents potential source vectors for zero-day
>>     exploits -- Common Weakness Enumeration (CWE)
>>     _https://cwe.mitre.org/_
>>
>>     *Vulnerability: *mistake in software that can be directly used by
>>     a hacker to gain access to a system or network; *Exposure:
>>     *configuration issue of a mistake in logic that allows
>>     unauthorized access or exploitation – Common Vulnerability and
>>     Exposure (CVE) _https://cve.mitre.org/_
>>
>>     *Exploit: *take advantage of a weakness (or multiple weaknesses)
>>     to achieve a _negative technical impact
>>     <https://capec.mitre.org/about/glossary.html>_ -- attack
>>     approaches from the set of known exploits are used in the Common
>>     Attack Pattern Enumeration and Classification (CAPEC)
>>     _https://capec.mitre.org <https://capec.mitre.org/>_
>>
>>     //
>>
>>     CVE, CWE, & CAPEC are part of the ITU-T CYBEX 1500 series & USG
>>     SCAP; so the distinction and relationship between weaknesses and
>>     vulnerabilities is already in standards. Let’s at least be
>>     consistent in our terminology with relevant standards.
>>
>>     Regards,
>>
>>        -Joe -
>>
>>     *Joe Jarzombek, **CSSLP***
>>
>>     Global Manager, Software Supply Chain Solutions
>>
>>     Email: joe.jarzombek at synopsys.com
>>     <mailto:joe.jarzombek at synopsys.com>  | Mobile: 703 627-4644
>>     <tel:%28703%29%20627-4644>
>>
>>     Synopsys Software Integrity Group  | www.synopsys.com/software
>>     <http://www.synopsys.com/software>
>>     cid:image002.jpg at 01D23376.09F8BF20
>>     <https://www.linkedin.com/company/synopsys-software-integrity>cid:image003.jpg at 01D23376.09F8BF20
>>     <https://www.facebook.com/SynopsysSoftwareIntegrity/>cid:image004.jpg at 01D23376.09F8BF20
>>     <https://twitter.com/sw_integrity>
>>
>>     SynopsysSiliconToSoftware
>>
>>     See my editorial in CrossTalk Sponsor’s Note on Supply Chain Risk
>>     in Critical Infrastructure http://bit.ly/2cb6OHU
>>
>>     See my presentation at AppSec USA --
>>
>>     *Software Supply Chain Lifecycle Management: Reducing Attack
>>     Vectors and Enabling Rugged DevOps*
>>
>>     _http://go.coverity.com/rs/157-LQW-289/images/SW%20SC20Lifecycle%20Management_AppSecUSA_13Oct2016.pdf
>>     <http://go.coverity.com/rs/157-LQW-289/images/SW%20SC20Lifecycle%20Management_AppSecUSA_13Oct2016.pdf>_
>>
>>     *From:*C-safe-secure-studygroup
>>     [mailto:c-safe-secure-studygroup-bounces at lists.trustable.io
>>     <mailto:c-safe-secure-studygroup-bounces at lists.trustable.io>] *On
>>     Behalf Of *Ian Hawkes via C-safe-secure-studygroup
>>     *Sent:* Friday, January 06, 2017 4:39 AM
>>     *To:* Robert Seacord <rcseacord at gmail.com>
>>     <mailto:rcseacord at gmail.com>; C Safety and Security Study Group
>>     Discussion <c-safe-secure-studygroup at lists.trustable.io>
>>     <mailto:c-safe-secure-studygroup at lists.trustable.io>
>>     *Subject:* Re: [C-safe-secure-studygroup] Static vs dynamic
>>     analysis: who cares?
>>
>>     I may well be wrong in my opinion here, so please forgive me if
>>     you think I am but it seems opinions are being expressed at the
>>     moment. The root topic of this group should be the C language as
>>     a language, not the testing (static, dynamic or otherwise). That
>>     should come as a by-product of the core activities.
>>
>>     I would like to suggest that the group focus on the core language
>>     and the vulnerabilities of the language that can manifest as
>>     safety or security issues.
>>
>>     The detailing of the issues can explain the what’s, whys and
>>     wherefores of the language vulnerability. The ability to detect
>>     the vulnerability can then be expressed as currently expected to
>>     be detected via static analysis or through dynamic analysis is a
>>     sub section of the vulnerability explanation. That way the
>>     document can be used as a guideline for how to use the C language
>>     in an application (architecture agnostic) and as a guidance as to
>>     where within the development process the vulnerabilities can be
>>     checked for. If new methods of detection come about that are
>>     universally accepted as sound, the guidelines section for that
>>     vulnerability can be up-revisioned to accommodate. This should
>>     not need to occur very often.
>>
>>     The safety processes such as ISO26262 (automotive) are not
>>     prescriptive about language but focus more on the process of
>>     using the languages. Security processes such as SDL are also very
>>     process focused.
>>
>>     Each industry and company that could be using this document will
>>     need to apply it to their individual development process that is
>>     steered / directed or flavoured by their specific industry
>>     guidelines / rules or mandates.
>>
>>     Keeping the document focused on the language and how the core
>>     elements can produce “issues” and how these issues may manifest
>>     themselves would allow tools vendors to concentrate on producing
>>     tools that are useful for businesses and allow the businesses to
>>     concentrate on how to apply the language to their products /
>>     processes / industries etc.
>>
>>     Polluting the document with too many objectives will not solve
>>     anything but will merely make the document less favoured by
>>     industry. Keep it simple and clear with the language at its heart
>>     and specific attributes around the elements of the language and
>>     you will find more businesses will make use of the document.
>>
>>     In my humble opinion that is what we should be trying to achieve
>>     here, a document that  is accepted by industry as a useful and
>>     applicable core guide so that it is accepted and used. Otherwise
>>     it is a document just for the sake of a document.
>>
>>     Mit freundlichen Grüßen \ Kind regards,
>>
>>     Ian Hawkes
>>
>>     *From:*C-safe-secure-studygroup
>>     [mailto:c-safe-secure-studygroup-bounces at lists.trustable.io
>>     <mailto:c-safe-secure-studygroup-bounces at lists.trustable.io>] *On
>>     Behalf Of *Robert Seacord via C-safe-secure-studygroup
>>     *Sent:* 05 January 2017 19:27
>>     *To:* Roberto Bagnara <bagnara at cs.unipr.it
>>     <mailto:bagnara at cs.unipr.it>>; C Safety and Security Study Group
>>     Discussion <c-safe-secure-studygroup at lists.trustable.io
>>     <mailto:c-safe-secure-studygroup at lists.trustable.io>>
>>     *Subject:* Re: [C-safe-secure-studygroup] Static vs dynamic
>>     analysis: who cares?
>>
>>     This appears to be a reasonable and a sound argument.  I think my
>>     biggest concern is as follow (so I would like to hear your views
>>     on this):
>>
>>     1. This group should involve analyzer vendors, as analyzer
>>     vendors are the ones who will conform to the standard, and our
>>     consequently our primary audience.
>>
>>     2. We are hoping that analyzer vendors will want to conform to
>>     the standard, because they will see this as a market opportunity.
>>
>>     3. Analyzer vendors could claim partial conformance but would
>>     prefer to claim conformance, as partial conformance is fairly
>>     meaningless (I'm sure I could claim partial conformance with grep).
>>
>>     4. Different analysis techniques are more or less capable of
>>     diagnosing different classes of problems.
>>
>>     5. Consequently, it will be difficult for a number of vendors,
>>     implementing different analysis techniques, to agree on a
>>     non-trivial rule set which they all feel they can enforce.
>>
>>     What do you think?
>>
>>     rCs
>>
>>     On Thu, Jan 5, 2017 at 1:51 PM, Roberto Bagnara via
>>     C-safe-secure-studygroup
>>     <c-safe-secure-studygroup at lists.trustable.io
>>     <mailto:c-safe-secure-studygroup at lists.trustable.io>> wrote:
>>
>>         Hi there.
>>
>>         I think that the current discussion on static analysis vs
>>         dynamic analysis
>>         is a distraction from (my perception) of the interests of the
>>         study group.
>>         In this writing, I will try to explain why.
>>
>>         Traditionally, a "static analysis" of a program is any
>>         analysis that
>>         can be conducted on the program without executing it.  And a
>>         "dynamic
>>         analysis" of a program is any analysis that is based on the
>>         observation
>>         of the behavior of a set of program executions.  This distinction
>>         should be taken with care.  The crucial point is what it means to
>>         execute a program:
>>
>>         1) the entire program can be executed on the intended target
>>         and the
>>            intended operational environment;
>>         2) parts of the program can be executed (with tests input
>>         generated in some
>>            way) on the intended target and the intended operational
>>         environment;
>>         3) a more or less related program (e.g., compiled for a
>>         target different
>>            from the intended target) can be executed, entirely or in
>>         parts,
>>            on an instrumented environment that is different from the
>>         intended
>>            operational environment;
>>         4) in abstract interpretation, (parts of) a program can be
>>         abstractly
>>            executed on a domain of data descriptions;
>>         5) (parts of) a program can be symbolically executed.
>>
>>         (Symbolic and concrete execution are mixed in so-called
>>         "concolic"
>>         execution, but I do not want to overcomplicate things.)
>>         I think there is no doubt that an analysis based on (1) is a
>>         dynamic
>>         analysis, that (4) and (5) are techniques used in various static
>>         analyzers, and that for (2) and (3) we would need more
>>         details in order
>>         to understand whether we are talking about a static or a
>>         dynamic analysis.
>>         The point, though, is: is this distinction really buying us
>>         anything?
>>
>>         I believe it is unfortunate that TS 17961:2016 mentions "static
>>         analysis" 12 times, sometimes in a way that seems to exclude
>>         tools
>>         that use techniques that are based on some sort of (abstract,
>>         symbolic,
>>         instrumented) execution of (parts of) the program.  My point
>>         is that,
>>         for the scope of this working group, we should not care about
>>         slippery
>>         distinctions about analysis techniques. What we should care about
>>         is automation.  To quote TS 17961:2016:
>>
>>             The purpose of this Technical Specification is to specify
>>             analyzable secure coding rules that can be automatically
>>         enforced
>>             to detect security flaws in C-conforming applications.
>>
>>         Automatically enforceable, this is the key.  And another
>>         point I like
>>         in TS 17961:2016 is in the "Scope" section:
>>
>>             This Technical Specification does not specify
>>               - the mechanism by which these rules are enforced [...]
>>               [...]
>>
>>         I propose to focus on these points and to forget about the
>>         distinction
>>         static analysys vs dynamic analysis. What we want are
>>         automatic tools
>>         where we push a button to start the analysis and, after a
>>         finite amount
>>         of time (perhaps enforced by the provision of a stop button which
>>         we can push when we want), they provide a set of results.
>>         Such results might contain false positives and false negatives:
>>         a conforming tool vendor will document, for each rule, whether
>>         false positives and/or false negatives are possible, so that
>>         we know
>>         whether compliance has been proved or if we have to do more
>>         work to
>>         prove it.  The number of false positives and false negatives
>>         is considered a quality of implementation issue by TS 17961:2016,
>>         the idea being that the market will do what it takes to
>>         promote the
>>         development of better tools.  Moreover, for a tool to be
>>         conforming
>>         it is enough that "[f]or each rule, the analyzer reports a
>>         diagnostic
>>         for at least one program that contains a violation of that rule."
>>
>>         Let us take an example:
>>
>>            5.2 Accessing freed memory [accfree]
>>            Rule
>>            After an allocated block of dynamic storage has been
>>         deallocated by
>>            a memory management function, the evaluation of any
>>         pointers into
>>            the freed memory, including being dereferenced or acting as an
>>            operand of an arithmetic operation, type cast, or
>>         right-hand side
>>            of an assignment, shall be diagnosed.
>>
>>         There are many techniques that allow to deal (to some extent)
>>         with this
>>         undecidable rule:
>>
>>         - abstract interpretation;
>>         - (symbolic) model checking;
>>         - random generation of test inputs;
>>         - constraint-based generation of test inputs;
>>         - language subsetting (e.g., MISRA C) and static analysis to
>>         prove
>>           the program is in the subset;
>>         - valgrind;
>>         - ...
>>
>>         Consider the use of valgrind: I run (a variant of) the program
>>         under valgrind for 10 minutes, by automatically generating inputs
>>         any way I like.  This technique is able to detect violations of
>>         [accfree] in a finite amount of time. It has false negatives
>>         (of course) but (depending on the variant and on the
>>         generated inputs)
>>         very few or no false positives.  All in all it may be better than
>>         many tools out there: why excluding it a priori?
>>
>>         My point is that our worry should be that the rule is
>>         specified in
>>         unambiguous terms, so that it is amenable to automatic
>>         cheking.  By
>>         this I mean, that the specification should be precise enough that
>>         consensus can be reached on whether a diagnostic given
>>         (resp., not
>>         given) by a tool has to be considered or not a false positive
>>         (resp.,
>>         a false negative).  This is not easy, but it is crucial.
>>         Distinguishing the automatic analysis technique(s) employed
>>         (static,
>>         dynamic, you-name-it) is, instead, irrelevant.
>>         Kind regards,
>>
>>            Roberto
>>
>>         --
>>            Prof. Roberto Bagnara
>>
>>         Applied Formal Methods Laboratory - University of Parma, Italy
>>         mailto:bagnara at cs.unipr.it <mailto:bagnara at cs.unipr.it>
>>                                     BUGSENG srl - http://bugseng.com
>>         <https://urldefense.proofpoint.com/v2/url?u=http-3A__bugseng.com&d=DgMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=uxVVCWR8UcUyeakFZXcD5bxiMa6Nx1hPouclYdJYz00&m=yidvjZb100BG4skYC17Wt_J5Iq86p4POFm34SpoYeJg&s=RnueYp3RHHKcLzinoBDl5AO3XZpXwfgALKh0m7IkO8c&e=>
>>                                    
>>         mailto:roberto.bagnara at bugseng.com
>>         <mailto:roberto.bagnara at bugseng.com>
>>
>>         _______________________________________________
>>         C-safe-secure-studygroup mailing list
>>         C-safe-secure-studygroup at lists.trustable.io
>>         <mailto:C-safe-secure-studygroup at lists.trustable.io>
>>         https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
>>         <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.trustable.io_cgi-2Dbin_mailman_listinfo_c-2Dsafe-2Dsecure-2Dstudygroup&d=DgMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=uxVVCWR8UcUyeakFZXcD5bxiMa6Nx1hPouclYdJYz00&m=yidvjZb100BG4skYC17Wt_J5Iq86p4POFm34SpoYeJg&s=w_k3nPkTzvBU2lM4oyU22B5hfpFIujBrGPtMQQrITTQ&e=>
>>
>>
>>
>>     _______________________________________________
>>     C-safe-secure-studygroup mailing list
>>     C-safe-secure-studygroup at lists.trustable.io
>>     <mailto:C-safe-secure-studygroup at lists.trustable.io>
>>     https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
>>     <https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup>
>     _______________________________________________
>     C-safe-secure-studygroup mailing list
>     C-safe-secure-studygroup at lists.trustable.io
>     <mailto:C-safe-secure-studygroup at lists.trustable.io>
>     https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
>     <https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 816 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0003.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 816 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0004.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 821 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0005.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 51878 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0001.png>


More information about the C-safe-secure-studygroup mailing list