[C-safe-secure-studygroup] Static vs dynamic analysis: who cares?
Barton Miller
bart at cs.wisc.edu
Fri Jan 6 17:10:57 UTC 2017
"Flaw" is certainly a reasonable synonym for weakness. (weakness ==
flaw == bug, for most folks)
I think that the key is to /not /use vulnerability where we are talking
about flaws in the code, unless it is associated with a specific exploit.
--bart
On 1/6/2017 11:04 AM, Robert Seacord wrote:
> On the subject(s) of vocabulary and free standards, it is preferable
> to use standard ISO definitions in ISO documents.
>
> The ISO/IEC 2382:2015(en) Information technology — Vocabulary is
> actually available for free from here:
>
> https://www.iso.org/obp/ui/#iso:std:iso-iec:2382:ed-1:v1:en
>
> If we can't find a definition we are looking for, we can add one. TS
> 17961 added the following definitions:
>
> https://gitlab.com/trustable/C_Safety_and_Security_Rules_Study_Group/wikis/terms-and-definitions
>
> Instead of "weaknesses", I think we used the following term:
>
>
> Security flaw
>
> Defect that poses a potential security risk
>
> We can certainly revisit this definition, as it is lacking form a
> safety perspective. We can also include definitions form normative
> references, which we do here:
> https://gitlab.com/trustable/C_Safety_and_Security_Rules_Study_Group/wikis/normative-references
>
> Another good source of definitions might be WG23:
> http://www.open-std.org/JTC1/SC22/WG23/docs/documents
>
> There are also some very lovely documents up there such as
>
> TR 24772-3 C Specific Language Vulnerabilities with Concurrency,
> submitted by David Keaton
> <http://www.open-std.org/JTC1/SC22/WG23/docs/ISO-IECJTC1-SC22-WG23_N0682-tr24772-3-post-meeting-46-with-concurrency-edits.docx>
>
>
> Which may be a great source for C specific language vulnerabilities
> with Concurrency (just a guess).
>
> If we would prefer to use terms defined outside of ISO/IEC documents,
> I think it is probably best that we just add them to our document. In
> fact, I just went ahead and added this to the wiki:
>
>
> Safety flaw
>
> Defect that poses a potential safety risk
>
> It was fun and easy! Everyone should try this. 8^) The benefit of
> changing things in the wiki (or wherever, pending instructions from
> Laurence and Chris Polin) is that if no one ever changes this it will
> end up in the International Standard. Stuff posted to the list doesn't
> end up anywhere except on the list.
>
> Thanks,
> rCs
>
> On Fri, Jan 6, 2017 at 11:26 AM, Barton Miller via
> C-safe-secure-studygroup <c-safe-secure-studygroup at lists.trustable.io
> <mailto:c-safe-secure-studygroup at lists.trustable.io>> wrote:
>
> +1
>
> Good to use the accepted terminology. (Though I have to admit,
> when speaking, I often use the wrong term!)
>
> --bart
>
>
> On 1/6/2017 10:15 AM, Joe Jarzombek via C-safe-secure-studygroup
> wrote:
>>
>> All –
>>
>> We are actually addressing ‘weaknesses’; not just
>> ‘vulnerabilities’ in the C language. Unless there is a specific
>> exploit identified that takes advantage of the flaw, it is only a
>> weakness (which could be a quality or security weakness). If an
>> exploit is identified that takes advantage of the flaw, then that
>> flaw is a vulnerability.
>>
>> Alternative statement: /The existence (even if only theoretical)
>> of an exploit designed to take advantage of a _weakness
>> <https://capec.mitre.org/about/glossary.html>_ (or multiple
>> weaknesses) and achieve a _negative technical impact
>> <https://capec.mitre.org/about/glossary.html>_ is what makes a
>> weakness a _vulnerability
>> <https://capec.mitre.org/about/glossary.html>_./
>>
>> Derived from definitions in ITU-T CYBEX 1500 series and the MITRE
>> websites for CWE, CVE and CAPEC:
>>
>> *Weakness: *mistake or flaw condition in architecture, design,
>> code, or process that, if left unaddressed, could under the
>> proper conditions contribute to a _cyber-enabled capability
>> <https://capec.mitre.org/about/glossary.html>_ being vulnerable
>> to exploitation; represents potential source vectors for zero-day
>> exploits -- Common Weakness Enumeration (CWE)
>> _https://cwe.mitre.org/_
>>
>> *Vulnerability: *mistake in software that can be directly used by
>> a hacker to gain access to a system or network; *Exposure:
>> *configuration issue of a mistake in logic that allows
>> unauthorized access or exploitation – Common Vulnerability and
>> Exposure (CVE) _https://cve.mitre.org/_
>>
>> *Exploit: *take advantage of a weakness (or multiple weaknesses)
>> to achieve a _negative technical impact
>> <https://capec.mitre.org/about/glossary.html>_ -- attack
>> approaches from the set of known exploits are used in the Common
>> Attack Pattern Enumeration and Classification (CAPEC)
>> _https://capec.mitre.org <https://capec.mitre.org/>_
>>
>> //
>>
>> CVE, CWE, & CAPEC are part of the ITU-T CYBEX 1500 series & USG
>> SCAP; so the distinction and relationship between weaknesses and
>> vulnerabilities is already in standards. Let’s at least be
>> consistent in our terminology with relevant standards.
>>
>> Regards,
>>
>> -Joe -
>>
>> *Joe Jarzombek, **CSSLP***
>>
>> Global Manager, Software Supply Chain Solutions
>>
>> Email: joe.jarzombek at synopsys.com
>> <mailto:joe.jarzombek at synopsys.com> | Mobile: 703 627-4644
>> <tel:%28703%29%20627-4644>
>>
>> Synopsys Software Integrity Group | www.synopsys.com/software
>> <http://www.synopsys.com/software>
>> cid:image002.jpg at 01D23376.09F8BF20
>> <https://www.linkedin.com/company/synopsys-software-integrity>cid:image003.jpg at 01D23376.09F8BF20
>> <https://www.facebook.com/SynopsysSoftwareIntegrity/>cid:image004.jpg at 01D23376.09F8BF20
>> <https://twitter.com/sw_integrity>
>>
>> SynopsysSiliconToSoftware
>>
>> See my editorial in CrossTalk Sponsor’s Note on Supply Chain Risk
>> in Critical Infrastructure http://bit.ly/2cb6OHU
>>
>> See my presentation at AppSec USA --
>>
>> *Software Supply Chain Lifecycle Management: Reducing Attack
>> Vectors and Enabling Rugged DevOps*
>>
>> _http://go.coverity.com/rs/157-LQW-289/images/SW%20SC20Lifecycle%20Management_AppSecUSA_13Oct2016.pdf
>> <http://go.coverity.com/rs/157-LQW-289/images/SW%20SC20Lifecycle%20Management_AppSecUSA_13Oct2016.pdf>_
>>
>> *From:*C-safe-secure-studygroup
>> [mailto:c-safe-secure-studygroup-bounces at lists.trustable.io
>> <mailto:c-safe-secure-studygroup-bounces at lists.trustable.io>] *On
>> Behalf Of *Ian Hawkes via C-safe-secure-studygroup
>> *Sent:* Friday, January 06, 2017 4:39 AM
>> *To:* Robert Seacord <rcseacord at gmail.com>
>> <mailto:rcseacord at gmail.com>; C Safety and Security Study Group
>> Discussion <c-safe-secure-studygroup at lists.trustable.io>
>> <mailto:c-safe-secure-studygroup at lists.trustable.io>
>> *Subject:* Re: [C-safe-secure-studygroup] Static vs dynamic
>> analysis: who cares?
>>
>> I may well be wrong in my opinion here, so please forgive me if
>> you think I am but it seems opinions are being expressed at the
>> moment. The root topic of this group should be the C language as
>> a language, not the testing (static, dynamic or otherwise). That
>> should come as a by-product of the core activities.
>>
>> I would like to suggest that the group focus on the core language
>> and the vulnerabilities of the language that can manifest as
>> safety or security issues.
>>
>> The detailing of the issues can explain the what’s, whys and
>> wherefores of the language vulnerability. The ability to detect
>> the vulnerability can then be expressed as currently expected to
>> be detected via static analysis or through dynamic analysis is a
>> sub section of the vulnerability explanation. That way the
>> document can be used as a guideline for how to use the C language
>> in an application (architecture agnostic) and as a guidance as to
>> where within the development process the vulnerabilities can be
>> checked for. If new methods of detection come about that are
>> universally accepted as sound, the guidelines section for that
>> vulnerability can be up-revisioned to accommodate. This should
>> not need to occur very often.
>>
>> The safety processes such as ISO26262 (automotive) are not
>> prescriptive about language but focus more on the process of
>> using the languages. Security processes such as SDL are also very
>> process focused.
>>
>> Each industry and company that could be using this document will
>> need to apply it to their individual development process that is
>> steered / directed or flavoured by their specific industry
>> guidelines / rules or mandates.
>>
>> Keeping the document focused on the language and how the core
>> elements can produce “issues” and how these issues may manifest
>> themselves would allow tools vendors to concentrate on producing
>> tools that are useful for businesses and allow the businesses to
>> concentrate on how to apply the language to their products /
>> processes / industries etc.
>>
>> Polluting the document with too many objectives will not solve
>> anything but will merely make the document less favoured by
>> industry. Keep it simple and clear with the language at its heart
>> and specific attributes around the elements of the language and
>> you will find more businesses will make use of the document.
>>
>> In my humble opinion that is what we should be trying to achieve
>> here, a document that is accepted by industry as a useful and
>> applicable core guide so that it is accepted and used. Otherwise
>> it is a document just for the sake of a document.
>>
>> Mit freundlichen Grüßen \ Kind regards,
>>
>> Ian Hawkes
>>
>> *From:*C-safe-secure-studygroup
>> [mailto:c-safe-secure-studygroup-bounces at lists.trustable.io
>> <mailto:c-safe-secure-studygroup-bounces at lists.trustable.io>] *On
>> Behalf Of *Robert Seacord via C-safe-secure-studygroup
>> *Sent:* 05 January 2017 19:27
>> *To:* Roberto Bagnara <bagnara at cs.unipr.it
>> <mailto:bagnara at cs.unipr.it>>; C Safety and Security Study Group
>> Discussion <c-safe-secure-studygroup at lists.trustable.io
>> <mailto:c-safe-secure-studygroup at lists.trustable.io>>
>> *Subject:* Re: [C-safe-secure-studygroup] Static vs dynamic
>> analysis: who cares?
>>
>> This appears to be a reasonable and a sound argument. I think my
>> biggest concern is as follow (so I would like to hear your views
>> on this):
>>
>> 1. This group should involve analyzer vendors, as analyzer
>> vendors are the ones who will conform to the standard, and our
>> consequently our primary audience.
>>
>> 2. We are hoping that analyzer vendors will want to conform to
>> the standard, because they will see this as a market opportunity.
>>
>> 3. Analyzer vendors could claim partial conformance but would
>> prefer to claim conformance, as partial conformance is fairly
>> meaningless (I'm sure I could claim partial conformance with grep).
>>
>> 4. Different analysis techniques are more or less capable of
>> diagnosing different classes of problems.
>>
>> 5. Consequently, it will be difficult for a number of vendors,
>> implementing different analysis techniques, to agree on a
>> non-trivial rule set which they all feel they can enforce.
>>
>> What do you think?
>>
>> rCs
>>
>> On Thu, Jan 5, 2017 at 1:51 PM, Roberto Bagnara via
>> C-safe-secure-studygroup
>> <c-safe-secure-studygroup at lists.trustable.io
>> <mailto:c-safe-secure-studygroup at lists.trustable.io>> wrote:
>>
>> Hi there.
>>
>> I think that the current discussion on static analysis vs
>> dynamic analysis
>> is a distraction from (my perception) of the interests of the
>> study group.
>> In this writing, I will try to explain why.
>>
>> Traditionally, a "static analysis" of a program is any
>> analysis that
>> can be conducted on the program without executing it. And a
>> "dynamic
>> analysis" of a program is any analysis that is based on the
>> observation
>> of the behavior of a set of program executions. This distinction
>> should be taken with care. The crucial point is what it means to
>> execute a program:
>>
>> 1) the entire program can be executed on the intended target
>> and the
>> intended operational environment;
>> 2) parts of the program can be executed (with tests input
>> generated in some
>> way) on the intended target and the intended operational
>> environment;
>> 3) a more or less related program (e.g., compiled for a
>> target different
>> from the intended target) can be executed, entirely or in
>> parts,
>> on an instrumented environment that is different from the
>> intended
>> operational environment;
>> 4) in abstract interpretation, (parts of) a program can be
>> abstractly
>> executed on a domain of data descriptions;
>> 5) (parts of) a program can be symbolically executed.
>>
>> (Symbolic and concrete execution are mixed in so-called
>> "concolic"
>> execution, but I do not want to overcomplicate things.)
>> I think there is no doubt that an analysis based on (1) is a
>> dynamic
>> analysis, that (4) and (5) are techniques used in various static
>> analyzers, and that for (2) and (3) we would need more
>> details in order
>> to understand whether we are talking about a static or a
>> dynamic analysis.
>> The point, though, is: is this distinction really buying us
>> anything?
>>
>> I believe it is unfortunate that TS 17961:2016 mentions "static
>> analysis" 12 times, sometimes in a way that seems to exclude
>> tools
>> that use techniques that are based on some sort of (abstract,
>> symbolic,
>> instrumented) execution of (parts of) the program. My point
>> is that,
>> for the scope of this working group, we should not care about
>> slippery
>> distinctions about analysis techniques. What we should care about
>> is automation. To quote TS 17961:2016:
>>
>> The purpose of this Technical Specification is to specify
>> analyzable secure coding rules that can be automatically
>> enforced
>> to detect security flaws in C-conforming applications.
>>
>> Automatically enforceable, this is the key. And another
>> point I like
>> in TS 17961:2016 is in the "Scope" section:
>>
>> This Technical Specification does not specify
>> - the mechanism by which these rules are enforced [...]
>> [...]
>>
>> I propose to focus on these points and to forget about the
>> distinction
>> static analysys vs dynamic analysis. What we want are
>> automatic tools
>> where we push a button to start the analysis and, after a
>> finite amount
>> of time (perhaps enforced by the provision of a stop button which
>> we can push when we want), they provide a set of results.
>> Such results might contain false positives and false negatives:
>> a conforming tool vendor will document, for each rule, whether
>> false positives and/or false negatives are possible, so that
>> we know
>> whether compliance has been proved or if we have to do more
>> work to
>> prove it. The number of false positives and false negatives
>> is considered a quality of implementation issue by TS 17961:2016,
>> the idea being that the market will do what it takes to
>> promote the
>> development of better tools. Moreover, for a tool to be
>> conforming
>> it is enough that "[f]or each rule, the analyzer reports a
>> diagnostic
>> for at least one program that contains a violation of that rule."
>>
>> Let us take an example:
>>
>> 5.2 Accessing freed memory [accfree]
>> Rule
>> After an allocated block of dynamic storage has been
>> deallocated by
>> a memory management function, the evaluation of any
>> pointers into
>> the freed memory, including being dereferenced or acting as an
>> operand of an arithmetic operation, type cast, or
>> right-hand side
>> of an assignment, shall be diagnosed.
>>
>> There are many techniques that allow to deal (to some extent)
>> with this
>> undecidable rule:
>>
>> - abstract interpretation;
>> - (symbolic) model checking;
>> - random generation of test inputs;
>> - constraint-based generation of test inputs;
>> - language subsetting (e.g., MISRA C) and static analysis to
>> prove
>> the program is in the subset;
>> - valgrind;
>> - ...
>>
>> Consider the use of valgrind: I run (a variant of) the program
>> under valgrind for 10 minutes, by automatically generating inputs
>> any way I like. This technique is able to detect violations of
>> [accfree] in a finite amount of time. It has false negatives
>> (of course) but (depending on the variant and on the
>> generated inputs)
>> very few or no false positives. All in all it may be better than
>> many tools out there: why excluding it a priori?
>>
>> My point is that our worry should be that the rule is
>> specified in
>> unambiguous terms, so that it is amenable to automatic
>> cheking. By
>> this I mean, that the specification should be precise enough that
>> consensus can be reached on whether a diagnostic given
>> (resp., not
>> given) by a tool has to be considered or not a false positive
>> (resp.,
>> a false negative). This is not easy, but it is crucial.
>> Distinguishing the automatic analysis technique(s) employed
>> (static,
>> dynamic, you-name-it) is, instead, irrelevant.
>> Kind regards,
>>
>> Roberto
>>
>> --
>> Prof. Roberto Bagnara
>>
>> Applied Formal Methods Laboratory - University of Parma, Italy
>> mailto:bagnara at cs.unipr.it <mailto:bagnara at cs.unipr.it>
>> BUGSENG srl - http://bugseng.com
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__bugseng.com&d=DgMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=uxVVCWR8UcUyeakFZXcD5bxiMa6Nx1hPouclYdJYz00&m=yidvjZb100BG4skYC17Wt_J5Iq86p4POFm34SpoYeJg&s=RnueYp3RHHKcLzinoBDl5AO3XZpXwfgALKh0m7IkO8c&e=>
>>
>> mailto:roberto.bagnara at bugseng.com
>> <mailto:roberto.bagnara at bugseng.com>
>>
>> _______________________________________________
>> C-safe-secure-studygroup mailing list
>> C-safe-secure-studygroup at lists.trustable.io
>> <mailto:C-safe-secure-studygroup at lists.trustable.io>
>> https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
>> <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.trustable.io_cgi-2Dbin_mailman_listinfo_c-2Dsafe-2Dsecure-2Dstudygroup&d=DgMGaQ&c=DPL6_X_6JkXFx7AXWqB0tg&r=uxVVCWR8UcUyeakFZXcD5bxiMa6Nx1hPouclYdJYz00&m=yidvjZb100BG4skYC17Wt_J5Iq86p4POFm34SpoYeJg&s=w_k3nPkTzvBU2lM4oyU22B5hfpFIujBrGPtMQQrITTQ&e=>
>>
>>
>>
>> _______________________________________________
>> C-safe-secure-studygroup mailing list
>> C-safe-secure-studygroup at lists.trustable.io
>> <mailto:C-safe-secure-studygroup at lists.trustable.io>
>> https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
>> <https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup>
> _______________________________________________
> C-safe-secure-studygroup mailing list
> C-safe-secure-studygroup at lists.trustable.io
> <mailto:C-safe-secure-studygroup at lists.trustable.io>
> https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup
> <https://lists.trustable.io/cgi-bin/mailman/listinfo/c-safe-secure-studygroup>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 816 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0003.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 816 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0004.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/jpeg
Size: 821 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0005.jpe>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 51878 bytes
Desc: not available
URL: <https://lists.trustable.io/pipermail/c-safe-secure-studygroup/attachments/20170106/b83454d3/attachment-0001.png>
More information about the C-safe-secure-studygroup
mailing list