unstructured/example-docs/test_evaluate_files/gold_standard_cct/Performance-Audit-Discussion.pdf.txt
shreyanid 6db663e7bb
refactor: separate click wrappers from core evaluation functionality (#1981)
### Summary
Click decorated functions cannot (properly) be called outside of the
click interface. This makes it difficult to reuse the setup
functionality in measure_text_edit_distance or
measure_element_type_accuracy. This PR removes the click decoration and
separates it into a wrapper function purely to execute the command.

### Technical Details
- Changed as suggested in [this StackOverflow
post](https://stackoverflow.com/questions/40091347/call-another-click-command-from-a-click-command)
response
- The locations of these now distinct functions are separate: the
`_command` click-decorated functions stay in ingest/evaluate.py, and the
core functions measure_text_edit_distance and
measure_element_type_accuracy are moved into the unstructured/metrics/
folder (which is a more logical location for them).
- Initial test added for measure_text_edit_distance

### Test
`sh ./test_unstructured_ingest/evaluation-metrics.sh text-extraction`
functionality is unchanged.

---------

Co-authored-by: ryannikolaidis <1208590+ryannikolaidis@users.noreply.github.com>
Co-authored-by: shreyanid <shreyanid@users.noreply.github.com>
Co-authored-by: Trevor Bossert <37596773+tabossert@users.noreply.github.com>
2023-11-07 19:54:22 +00:00

205 lines
11 KiB
Plaintext
Raw Blame History

This file contains invisible Unicode characters

This file contains invisible Unicode characters that are indistinguishable to humans but may be processed differently by a computer. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

Page 1
The introductory chapter of Government Auditing Standards (GAGAS)1
outlines five concepts describing how public officials are to provide
functions and services: effectively, efficiently, economically, ethically, and
equitably. When planning, gathering and assessing evidence, and
reporting audit results, auditors may focus on one or more of these
concepts. The following discussion is intended to assist auditors when
developing audit objectives for performance audits of government
programs and activities.2
This discussion is designed to help auditors understand
and apply the concepts cited above for performance audits
conducted in accordance with GAGAS. This discussion
does not contain requirements, does not amend GAGAS,
and is not considered interpretive guidance, as defined in
chapter 2 of GAGAS.
Paragraph 1.02:
The concept of accountability for use of public resources
and government authority is key to our nations governing
processes. Management and officials entrusted with public
resources are responsible for carrying out public functions
and providing service to the public effectively, efficiently,
economically, ethically, and equitably within the context
of the statutory boundaries of the specific government
program. [Emphasis added.]
Paragraph 1.03:
As reflected in applicable laws, regulations, agreements,
and standards, management and officials of government
programs are responsible for providing reliable, useful, and
timely information for transparency and accountability of
these programs and their operations. Legislators, oversight
1GAO, Government Auditing Standards: 2018 Revision, GAO-21-368G (Washington,
D.C.: April 2021)
2The concepts cited may also be applicable to other GAGAS engagements, based on the
auditors judgments. This discussion is limited to considering these concepts in
performance audits.
GAGAS Performance Audits: Discussion of
Concepts to Consider When Auditing Public
Functions and Services
GAGAS Paragraphs
Page 2
bodies, those charged with governance, and the public
need to know whether (1) management and officials
manage government resources and use their authority
properly and in compliance with laws and regulations; (2)
government programs are achieving their objectives and
desired outcomes; and (3) government services are
provided effectively, efficiently, economically, ethically,
and equitably. [Emphasis added.]
Government administration best serves the collective interest of the public
when it is effective, efficient, economical, ethical, and equitable. Auditors
help inform legislators, oversight bodies, those charged with governance,
and the public about whether public services are being provided
consistent with these concepts. Government auditing can contribute to
accountability and can help improve government administration by
identifying deficiencies and recommending enhancements to achieve
effective, efficient, economical, ethical, and equitable outcomes, when
appropriate within the context of the audit objectives. As such, it is
important for auditors to understand the concepts below as they relate to
administering government programs or activities and how they can
assess or address these expectations of government performance in
conducting their performance audits.
The examples that follow the discussion of each concept illustrate the
distinctions between these concepts. In a performance audit, it is
common practice to incorporate more than one of these concepts when
conducting the audit.
The administration of a government program or activity is effective when
it achieves the intended results. A performance audit that focuses on the
effectiveness of a program or activity seeks to establish a cause-andeffect relationship between the operation of the program or activity and
achieving its stated objectives. Achieving the objectives does not
guarantee that the program or activity was effective unless the auditors
can establish that the program or activity caused, or contributed to, the
desired outcome.
Example: In a performance audit examining how effective a
housing voucher program was in achieving its goal of improving
economic outcomes for recipients, auditors may determine
whether receiving housing vouchers led to better subsequent
economic outcomes for recipients than those of similarly situated
individuals who did not receive vouchers.
Discussion
Effective
Page 3
Example: In a performance audit assessing the effectiveness of
an after-school program targeted at helping students improve their
reading proficiency, auditors may examine the extent to which
participants reading levels improved relative to baseline data from
before they joined the program.
The administration of a government program or activity is efficient when
it gets the most value from available resources. When a performance
audit focuses on efficiency, auditors examine whether the resources used
to administer a program or activity have been put to optimal or
satisfactory use, or whether the same or similar results could have been
achieved more timely or with fewer resources.
Example: In a performance audit assessing a disaster relief
agencys mobilization of resources to respond to a disaster,
auditors may assess the disaster relief agencys timeliness in
providing relief compared to its own previous performance or the
performance of other similarly situated agencies that have
responded to comparable disasters.
Example: In a performance audit assessing a consumer protection
agencys response to consumer complaints, auditors may assess
whether the agencys efforts to streamline its processes resulted
in improved timely resolution of complaints.
Example: In a performance audit assessing the time a state needs
to process unemployment benefits targeted at helping those in
need, auditors may assess how long the process takes from
receipt of the unemployment application to the applicants receipt
of the benefit, including steps such as verifying required
information.
The administration of a government program or activity is economical
when it minimizes the costs of resources used in performing its functions
while meeting timeliness and quality considerations for those resources.
When auditing economy, auditors primarily focus on the costs of inputs
rather than on the outcomes achieved.
Example: In a performance audit examining an agencys
international travel expenses, in addition to assessing the design
of internal controls and compliance with expense guidelines,
auditors may test whether, for a sample of trips, bookings of
Efficient
Economical
Page 4
equivalent airline tickets and hotel rooms could be found at a
lower cost.
Example: In a performance audit assessing an agencys
acquisition practices, auditors may examine whether the agencys
decisions regarding purchasing, leasing, or reimbursing
employees for the costs of acquiring various supplies or
equipment achieved the lowest cost while meeting applicable
requirements.
The administration of a government program or activity is ethical when it
advances the collective interest of the public rather than private gain and
is conducted with honesty, integrity, and impartiality. Laws and
regulations often specify rules of ethical conduct. Therefore, audits
examining the ethical administration of a program or activity may involve
assessing compliance with such laws and regulations. Fraud in
administering a government program or activity betrays the public trust
and is, by definition, unethical. In addition, auditors may identify instances
of unethical conduct that result in waste and abuse during testing of
internal controls as part of a performance audit.
Example: In a performance audit assessing agency officials
compliance with conflict-of-interest requirements, auditors may
compare a sample of financial disclosure reports filed against
requirements in statute or regulation.
Example: In a performance audit assessing potential regulatory
capture related to a particular industry, auditors may assess the
extent to which the regulatory agency has sufficient controls to
reasonably assure its employees independence from the entities
subject to the agencys regulation.
Example: In a performance audit assessing an offices policies
and procedures for purchase cards, auditors testing of the
programs controls to identify deficiencies may identify fraud,
waste, or abuse in its administration.
The administration of a government program or activity is equitable when
it consistently serves members of the public, distributes public services,
and implements public policy in a manner that promotes fairness, justice,
and equality. Auditing whether the administration of a government
program or activity is equitable may include assessing the
Ethical
Equitable
Page 5
• equality of access to and provision of services;
• procedural fairness and equal treatment of individuals in
government programs and policies;
• causes of disparate outcomes;
• or distributional impacts of public policies, programs, resources,
and services.
Disaggregating data by social groups or communities that share a
particular characteristic (e.g., gender, race, ethnicity, age, or income)
can help illuminate differences. Reporting on such differences, when
appropriate within the context of the audit objectives, can increase
understanding of the effects of policies and programs on issues of
equity.
Example: In a performance audit assessing the granting of
waivers from particular requirements, auditors may use
disaggregated data about waiver recipients to assess whether
different groups or communities were treated fairly and equally in
the process.
Example: In a performance audit assessing a grant program
aimed at expanding internet access, auditors may assess the
extent to which formulas, criteria, or other factors (such as
matching funds or capital requirements) considered in the
distribution of grant funds may be to the specific advantage or
disadvantage of certain groups, regions, or communities, thereby
causing inequities.
Example: In a performance audit assessing scholarship outcomes
in higher education programs, auditors may report on the
distribution of scholarships by race, gender identity, and income to
illuminate potential disparities among scholarship recipients.
These concepts may overlap. For example, efficiency may also be a
component of effectiveness. Similarly, when appropriate within the
context of the program and audit objectives, auditors may disaggregate
the results of performance audits that focus on efficiency or effectiveness
Page 6
issues to illuminate inequities in program administration or in distribution
of public services.
While all of these concepts are important to administering government
programs responsibly, it is up to the professional judgment of the auditors
to determine the specific concepts that are relevant in conducting the
performance audit and reporting the results. Auditors professional
judgments are informed by, among other things, the needs of the users of
the audit reports; the nature, context, and objectives of the program or
activity under audit; and the public interest.
To view the current Yellow Book, visit https://www.gao.gov/yellowbook.
For technical assistance, call (202) 512-9535 or email
yellowbook@gao.gov.
For More Information