Grading the Government’s
Data Publication Practices

by Jim Harper

Executive Summary

Barack Obama promised transparency and open government when he campaigned for president in 2008, and he took office aiming to deliver it. Today, the federal government is not transparent, and government transparency has not improved materially since the beginning of President Obama’s administration. This is not due to lack of interest or effort, though. Along with meeting political forces greater than his promises, the Obama transparency tailspin was a product of failure to apprehend what transparency is and how it is produced.

A variety of good data publication practices can help produce government transparency: authoritative sourcing, availability, machine-discoverability, and machine-readability. The Cato Institute has modeled what data the government should publish in the areas of legislative process and budgeting, spending, and appropriating. The administration and the Congress both receive fairly low marks under systematic examination of their data publication practices.

Between the Obama administration and House Republicans, the former, starting from a low transparency baseline, made extravagant promises and put significant effort into the project of government transparency. It has not been a success. House Republicans, who manage a far smaller segment of the government, started from a higher transparency baseline, made modest promises, and have taken limited steps to execute on those promises. President Obama lags behind House Republicans, but both have a long way to go.

Jim Harper is director of information policy studies at the Cato Institute and the webmaster of WashingtonWatch.com.

Introduction

As a campaigner in 2008, President Obama promised voters hope, change, and transparency.¹ Within minutes of his taking office on January 20, 2009, in fact, the Whitehouse.gov website declared: “President Obama has committed to making his administration the most open and transparent in history.”² His first presidential memorandum, issued the next day, was entitled “Transparency and Open Government.” It declared:

My Administration is committed to creating an unprecedented level of openness in Government. We will work together to ensure the public trust and establish a system of transparency, public participation, and collaboration. Openness will strengthen our democracy and promote efficiency and effectiveness in Government.³

The road to government transparency is long. Nearly four years later, few would argue that American democracy has materially strengthened, or that the government is any more effective and efficient, due to forward strides in transparency and openness. Indeed, the administration has come under fire recently—as every administration does, it seems—for significant transparency failings.

Freedom of Information Act (FOIA) policy is an example. In its early days, the Obama administration committed to improving the government’s FOIA practices. In March 2009 Attorney General Eric Holder issued a widely lauded memorandum ordering improvements in FOIA compliance.⁴ But this September, Bloomberg news reported on its test of the Obama Administration’s commitment to transparency under FOIA. Bloomberg found that 19 of 20 cabinet-level agencies disobeyed the public disclosure law when it asked for information about the cost of agency leaders’ travel. Just 8 of 57 federal agencies met Bloomberg’s request for documents within the 20-day disclosure window required by the act.⁵

President Obama’s campaign promise to post laws to the White House website for five days of public comment before he signed them went virtually ignored by the White House in the first year of his administration. Only recently has he reached two-thirds compliance with the “Sunlight Before Signing” promise, and this is because of the multitude of bills Congress passes to rename post offices and such. More important bills are often given less than the promised five days’ sunlight.⁶

There was no lack of effort or creativity around data transparency at the outset of the Obama administration. In May 2009 White House officials announced on the new Open Government Initiative blog that they would elicit the public’s input into the formulation of its transparency policies. In a meta-transparency flourish, the public was invited to join in with the brainstorming, discussion, and drafting of the government’s policies.⁷

The conspicuously transparent, participatory, and collaborative process contributed something, evidently, to an “Open Government Directive,” issued in December 2009 by Office of Management and Budget head Peter Orszag.⁸ Its clear focus was to give the public access to data. The directive ordered agencies to publish within 45 days at least three previously unavailable “high-value data sets” online in an open format and to register them with the federal government’s data portal, Data.gov. Each agency was to create an “Open Government Webpage” as a gateway to agency activities related to the Open Government Directive.

Many, many of President Obama’s transparency promises went by the wayside. His guarantee that health care legislation would be negotiated “around a big table” and televised on C-SPAN was quite nearly the opposite of what occurred.⁹ People are free to observe whether it is political immaturity, idealism, or dishonesty that prompted transparency promises of this kind. Whatever the case, history may show that the “high-value data set” challenge was where the Obama Administration’s data transparency effort began its tailspin.

Celebrated though it is, transparency is not a well-defined concept, and the administration’s most concerted effort to deliver it missed the mark. The reason is that the definition of “high-value data set” it adopted was hopelessly vague:

High-value information is information that can be used to increase agency accountability and responsiveness; improve public knowledge of the agency and its operations; further the core mission of the agency; create economic opportunity; or respond to need and demand as identified through public consultation.

Essentially anything agencies wanted to publish they could publish claiming “high value” for it.

Agencies “adopted a passive-aggressive attitude” toward the Data.gov effort, according to political scientist Alon Peled.¹⁰ They technically complied with the requirements of the Open Government Memorandum, but did not select data that the public valued.

The Open Government Directive allowed agencies to exploit a subtle “shift in vocabulary” in the area of open government. They diverted the project away from the core government transparency that the public found so attractive about President Obama’s campaign claims. “The term ‘open government data’ might refer to data that makes the government as a whole more open (that is, more publicly accountable),” write Harlan Yu and David Robinson, “or instead might refer to politically neutral public sector disclosures that are easy to reuse, even if they have nothing to do with public accountability.”¹¹

The Agriculture department published data about the race, ethnicity, and gender of farm operators, for example, rather than about the funds it spent to collect that kind of information. An informal Cato Institute study examining agencies’ “high-value” data feeds found, “almost uniformly, the agencies came up with interesting data—but ‘interesting’ is in the eye of the beholder. And interesting data collected by an agency doesn’t necessarily give the insight into government we were looking for.”¹²

Genuinely high-value data for purposes of government transparency would provide insight in three areas not found in many of the early Data.gov feeds. True high-value data would be about government entities’ management, deliberations, or results.¹³

“Open data can be a powerful force for public accountability,” write Yu and Robinson, “It can make existing information easier to analyze, process, and combine than ever before, allowing a new level of public scrutiny.”¹⁴ This is undoubtedly true, and Americans have experienced vastly increased access to information in so many walks of life—shopping, news-gathering, and investments, to name just three. Data-starved public oversight of government appears sorely lacking in comparison.

In September a new transparency-related international initiative took center stage for the administration, the Open Government Partnership (OGP).¹⁵ This “multilateral initiative” was created “to promote transparency, fight corruption, strengthen accountability, and empower citizens.”¹⁶ Participating countries pledged “to undertake meaningful new steps as part of a concrete action plan, developed and implemented in close consultation with their citizens.” The OGP website touts a panoply of meetings, plans, and social media outreach efforts, and a recent graphic displayed on the home page said in bold letters, “From Commitment to Action.” Its authors probably have no sense of the irony in that declaration. Significant actions, after all, announce themselves.

Nothing about the OGP is harmful, and it may produce genuine gains for openness in participating countries. However, it has not produced, and does not hold out, the fundamental change—data-oriented change—that was at the heart of President Obama’s campaign promises.

The Obama administration is not the only actor on the federal stage, of course. House Republicans made transparency promises of their own in the course of their campaign to retake control of the House of Representatives, which they did in 2011.

“The lack of transparency in Congress has been a problem for generations, under majorities Republican and Democrat alike,” said aspiring House speaker John Boehner (R-OH) in late 2009. “But with the advent of the Internet, it’s time for this to change.”¹⁷

Since 1995, the Library of Congress’s THOMAS website has published information, sometimes in the form of useful data, about Congress and its activities. Upon taking control of the House for the first time in 40 years, the Republican leadership of the 104th Congress directed the Library of Congress to make federal legislative information freely available to the public. The offerings on the site now include bills, resolutions, activity in Congress, the Congressional Record, schedules, calendars, committee information, the president’s nominations, and treaties.¹⁸

In an attempt to improve the availability of key information, at the beginning of the 112th Congress the House instituted a rule—not always complied with—that bills should be posted online for three calendar days before receiving a vote on the House floor.¹⁹ The House followed up by creating a site at data.house.gov where such bills are posted. In February 2012 the House Committee on Administration held a day-long conference on legislative data,²⁰ evidence of continuing interest and of plans to move forward. And in September, the Library of Congress debuted beta.congress.gov, which is slated to be the repository for legislative data that ultimately replaces the THOMAS website.²¹

Between the Obama administration and House Republicans, the former, starting from a low transparency baseline, made extravagant promises and put significant effort into the project of government transparency. It has not been a success. House Republicans, who manage a far smaller segment of the government, started from a higher transparency baseline, made modest promises, and have taken limited steps to execute those promises.

The transparency problem is far from solved, of course. The information that the public would use to increase their oversight and participation is still largely inaccessible. The Republican House may be ahead, but both the administration and Congress score poorly under systematic examination of their data publication practices.

The Data that Would Make for Transparent Government

It was not disinterest that caused the Obama administration transparency effort to fade. Arguably, it was the failure of the transparency community to ask clearly for what it wants: good data about the deliberations, management, and results of government entities and agencies. So in January 2011 the Cato Institute began working with a wide variety of groups and advisers to “model” governmental processes as data and then to prescribe how this data should be published.

Data modeling is arcane stuff, but it is worth understanding here at the dawn of the Information Age. “Data” is collected abstract representations of things in the world. We use the number “3,” for example, to reduce a quantity of things to an abstract, useful form—an item of data. Because clerks can use numbers to list the quantities of fruits and vegetables on hand, store managers can effectively carry out their purchasing, pricing, and selling instead of spending all of their time checking for themselves how much of everything there is. Data makes everything in life a little easier and more efficient for everyone.

Legislative and budgetary processes are not a grocery store’s produce department, of course. They are complex activities involving many actors, organizations, and steps. The Cato Institute’s modeling of these processes reduced everything to “entities,” each having various “properties.” The entities and their properties describe the things in legislative and budgetary processes and the logical relationships among them, like members of Congress, the bills they introduce, hearings on the bills, amendments, votes, and so on. The “entity” and “property” terminology corresponds with usage in the world of data management, it is used to make coding easier for people in that field, and it helps to resolve ambiguities in translating governmental processes into useful data. The modeling was restricted to formal parts of the processes, excluding, for example, the varied organizations that try to exert influence, informal communications among members of Congress, and so on.

The project also loosely defined several “markup types,” guides for how documents that come out of the legislative process should be structured and published to maximize their utility. The models and markup types are discussed in a pair of Cato@Liberty blog posts that also issued preliminary grades on the quality of data publication about the entities.²²

Next, the project examined the publication methods that allow data to reach its highest and best use. Four key data practices that support government transparency emerged. Documented in a Cato Institute Briefing Paper entitled “Publication Practices for Transparent Government,”²³ those practices are authoritative sourcing, availability, machine-discoverability, and machine-readability.

Authoritative sourcing means producing data as near to its original source and time as possible, so that the public uniformly comes to rely on the best sources of data. The second transparent data practice, availability, entails consistency and confidence in data, including permanence, completeness, and good updating practices.

The third transparent data practice, machine-discoverability, occurs when information is arranged so that a computer can discover the data and follow linkages among it. Machine-discoverability exists when data is presented consistently with a host of customs about how data is identified and referenced, the naming of documents and files, the protocols for communicating data, and the organization of data within files.

The fourth transparent data practice, machine-readability, is the heart of transparency because it allows the many meanings of data to be discovered. Machine-readable data is logically structured so that computers can automatically generate the myriad stories that the data has to tell and put it to the hundreds of uses the public would make of it in government oversight. A common and popular language for structuring and containing data is called XML, or eX-tensible Markup Language, which is a relative of HTML (hypertext markup language), the language that underlies the World Wide Web.

Beginning in September 2011 the project graded how well Congress and the administration publish data about the key entities in the processes they oversee. Congress is responsible for data pertaining to the legislative process, of course. The administration has the bulk of the responsibility for budget-related data (except for the congressional budgets and appropriations). These grades are available in a pair of Cato@Liberty blog posts²⁴ and in Appendixes A and B.

With the experience of the past year, the project returned to grading in September 2012. With input from staff at GovTrack. us, the National Priorities Project, OMB Watch, and the Sunlight Foundation (their endorsement of the grades not implied by their assistance), we assessed how well data is now published. The grades presented in Figures 1 and 2 are largely consistent with the prior year—little changed between the two grading periods—but there were some changes in grades in both directions due to improvements in publication, discovery of data sources by our panel of graders, and heightened expectations. “Incompletes” given in the first year of grading became Fs in some cases and Ds in others.

It is important to highlight that grades are a lagging indicator. Transparency is not just a product of good data publication, but also of the society’s ability to digest and use information. Once data feeds are published, it takes a little while for the community of users to find them and make use of them. A new web site dedicated to congressional information, beta.congress.gov, will undoubtedly improve data transparency and the grades for data it publishes, assuming it lives up to expectations.

Government transparency is a widely agreed-upon value, sought after as a means toward various ends. Libertarians and conservatives support transparency because of their belief that it will expose waste and bloat in government. If the public understands the workings and failings of government better, the demand for government solutions will fall and democracy will produce more libertarian outcomes. American liberals and progressives support transparency because they believe it will validate and strengthen government programs. Transparency will root out corruption and produce better outcomes, winning the public’s affection and support for government.

Though the goals may differ, pan-ideological agreement on transparency can remain. Libertarians should not prefer large government programs that are failing. If transparency makes government work better, that is preferable to government working poorly. If the libertarian vision prevails, on the other hand, and transparency produces demand for less government and greater private authority, that will be a result of democratic decisionmaking that liberals and progressives should respect and honor.

With that, here are the major entities in the legislative process and in budgeting, appropriating, and spending; the grades that reflect the quality of the data published about them; and a discussion of both.

Publication Practices for Transparent Government: Rating Congress

House Membership: C-
Senate Membership: A-

It would seem simple enough to publish data about who holds office in the House of Representatives and Senate, and it is. There are problems with the way the data is published, though, which the House and Senate could easily remedy.

On the positive side—and this is not to be discounted—there is a thing called the “Biographical Directory of the United States Congress,” a compendium of information about all present and former members of the U.S. Congress (as well as the Continental Congress), including delegates and resident commissioners. The “Bioguide” website at bioguide.congress.gov is a great resource for searching out historical information.

But there is little sign that Bioguide is Congress’s repository of record, and it is little known by users, giving it lower authority marks than it should have. Some look to the House and Senate websites and beta. congress.gov for information about federal representatives, splitting authority among websites, rather than one established and agreed upon resource.

Bioguide scores highly on availability—we know of no problems with up-time or completeness (though it could use quicker updating when new members are elected). Bioguide is not structured for discoverability, though. Most people have not seen it, because search engines are not finding it.

Bioguide does a good thing in terms of machine readability, though. It assigns a unique ID to each of the people in its database. This is the first, basic step in making data useful for computers, and the Bioguide ID should probably be the standard for machine identification of elected officials wherever they are referred to in data. Unfortunately, the biographical content in Bioguide is not machine-readable.

As noted above, the other ways of learning about House and Senate membership are ad hoc. The Government Printing Office has a “Guide to House and Senate Members” at http://memberguide.gpo.gov/ that duplicates information found elsewhere. The House website presents a list of members along with district information, party affiliation, and so on, in HTML format (http://www.house.gov/representatives/), and beta. congress.gov does as well (http://beta.congress.gov/members/). Someone who wants a complete dataset must collect data from these sources using a computer program to scrape the data and through manual curation. The HTML presentations do not break out key information in ways useful for computers. The Senate membership page,²⁵ on the other hand, includes a link to an XML representation that is machine readable. That is the reason why the Senate scores so well compared to the House.

Much more information about our representatives flows to the public via representatives’ individual websites. These are nonauthoritative websites that search engine spidering combines to use as a record of the Congress’s membership. They are available and discoverable, again because of that prime house.gov and senate.gov real estate. But they only reveal data about the membership of Congress incidentally to communicating the press releases, photos, and announcements that representatives want to have online.

It is a narrow point, but there should be one and only one authoritative, well-published source of information about House and Senate membership from which all others flow. The variety of sources that exist combine to give Congress pretty good grades on publishing information about who represents Americans in Washington, but improving in this area is a simple matter of coordinated House and Senate efforts.

Committees and Subcommittees: C-

Like Americans’ representation in Congress, lists of committees, their membership, and jurisdiction should be an easy lift. But it is not as easy as it should be to learn about the committees to which Congress delegates much of its work and the subcommittees to which the work gets further distributed.

The Senate has committee names and URLs prominently available on its main website.²⁶ The House does, too, at http://house.gov/committees/. But neither page offers machine-readable information about committees and committee assignments.

The Senate has a nice list of committee assignments, again, though, not machine-readable. The House requires visitors to click through to each committee’s web page to research what they do and who serves on them. For that, you’d go to individual committee websites, each one different from the others. There is an authoritative list of House committees with unique identifiers,²⁷ but it’s published as a PDF, and it is not clear that it is used elsewhere for referring to committees.

Without a recognized place to go to get data about committees, this area suffers from lacking authority. To the extent there are data, availability is not a problem, but machine-discoverability suffers for having each committee publish distinctly, in formats like HTML, who their members are, who their leaders are, and what their jurisdiction is.

With the data scattered about this way, the Internet can’t really see it. More prominence, including data such as subcommittees and jurisdiction, and use of a recognized set of standard identifiers would take this resource a long way.

Until committee data are centrally published using standard identifiers (for both committees and their members), machine-readability will be very low. The Internet makes sense of congressional committees as best it can, but a whole lot of organizing and centralizing—with a definitive, always-current, and machine-readable record of committees, their memberships, and their jurisdictions—would create a lot of clarity in this area with a minimum of effort.

Meetings of House, Senate, and Committees—House: B/Senate: B

When the House, the Senate, committees, and subcommittees have their meetings, the business of the people is being done. Can the public learn easily about what meetings are happening, where they are happening, when they are happening, and what they are about? It depends on which side of the Capitol you’re on.

The Senate is pretty good about publishing notices of committee meetings. From a webpage with meeting notices listed on it,²⁸ there is a link to an XML version of the data to automatically inform the public.

If a particular issue is under consideration in a Senate committee meeting, this is a way for the public to learn about it. This is authoritative, it is available, it is machine-discoverable, and has some machine-readable features. That means any application, website, researcher, or reporter can quickly use these data to generate more—and more useful—information about Congress.

The House does not have anything similar for committee meetings. To learn about those meetings, one has to scroll through page after page of committee announcements or calendars. Insiders subscribe to paid services. The House can catch up with the Senate in this area.

Where the House excels and the Senate lags is in notice about what will be considered on the floor. The House made great strides with the institution of docs.house.gov, which displays legislation heading for the floor. This allows any visitor, and various websites and services, to focus their attention on the nation’s business for the week.

Credit is due the House for establishing this resource and using it to inform the public using authoritative, available, and machine-discoverable and -readable data. This is an area where the Senate has the catching up to do.

For different reasons, the House and Senate both garner Bs. Were they to copy the best of each other, they would both have As.

Meeting Records: D-

There is a lot of work to do before meeting records can be called transparent. The Congressional Record is the authoritative record of what transpires on the House and Senate floors, but nothing similar reveals the content of committee meetings. Those meeting records are produced after much delay—sometimes an incredibly long delay—by the committees themselves. These records are obscure, and they are not being published in ways that make things easy for computers to find and comprehend.

In addition, the Congressional Record doesn’t have the machine-discoverable publication or machine-readable structure that it could and should. Giving unique, consistent IDs in the Record to members of Congress, to bills, and other regular subjects of this publication would go a long way to improving it. The same would improve transcripts of committee meetings.

Another form of meeting record exists: videos. These have yet to be standardized, organized, and published in a reliable and uniform way, but the HouseLive site (http://houselive.gov/) is a significant step in the right direction. It will be of greater use when it can integrate with other records of Congress. Real-time flagging of members and key subjects of debate in the video stream would be a great improvement in transparency. Setting video and video meta-data standards for use by both Houses of Congress, by committees, and by subcommittees would improve things dramatically.

House video is a bright spot in a very dark field, but both will shine brighter in time. When the surrounding information environment has improved to educate the public about goings-on in Congress in real time, the demand for and usefulness of video will increase.

Committee Reports: C+

Committee reports are important parts of the legislative process, documenting the findings and recommendations that committees report to the full House and Senate. They do see publication on the most authoritative resource for committee reports, the Library of Congress’s THOMAS system. They are also published by the Government Printing Office.²⁹ The GPO’s Federal Digital System (FDsys) is relatively new and is meant to improve systematic access to government documents, but it has not become recognized as an authoritative source for many of those documents.

Because of the sources through which they are published, committee reports are somewhat machine-discoverable, but without good semantic information embedded in them, committee reports are barely visible to the Internet.

Rather than publication in HTML and PDF, committee reports should be published fully marked up with the array of signals that reveal what bills, statutes, and agencies they deal with, as well as authorizations and appropriations, so that the Internet can discover and make use of these documents.

Bills: B-

Bills are a “pretty-good-news” story in legislative transparency. Most are promptly published. It would be better, of course, if they were all immediately published at the moment they were introduced, and if both the House and Senate published last-minute, omnibus bills before debating and voting on them.

A small gap in authority exists around bills. Some people look to the Library of Congress and the THOMAS site, and now beta.congress.gov, for bill information. Others look to the Government Printing Office. Which is the authority for bill content? This issue has not caused many problems so far. Once published, bill information remains available, which is good.

Publication of bills in HTML on the THOMAS site makes them reasonably machine-discoverable. Witness the fact that searching for a bill will often turn up the version at that source.

Where bills could improve some is in their machine-readability. Some information such as sponsorship and U.S. code references is present in the bills that are published in XML, and nearly all bills are now published in XML, which is great. Much more information should be published machine-readably in bills, though, such as references to agencies and programs, to states or localities, to authorizations and appropriations, and so on, referred to using standard identifiers.

With the work that the THOMAS system does to gather information in one place, bill data are good. This is relative to other, less-well-published data, though. There is yet room for improvement.

Amendments: F

Amendments are not the good-news story that bills are. They are “barely available,” says Eric Mill of the Sunlight Foundation. “Given that amendments (especially in the Senate) can be as large and important as original legislation, this is an egregious oversight.”

With a few exceptions, amendments are hard to track in any systematic way. When bills come to the House and Senate floors, amendment text is often available, but amendments are often plopped somewhere in the middle of the Congressional Record without any reliable, understood, machine-readable connection to the underlying legislation. It is very hard to see how amendments affect the bills they would change.

In committees, the story is quite a bit worse. Committee amendments are almost completely opaque. There is almost no publication of amendments at all—certainly not amendments that have been withdrawn or defeated. Some major revisions in process are due if committee amendments are going to see the light of day as they should.

Motions: F

When the House, the Senate, or a committee is going to take some kind of action, it does so on the basis of a motion. If the public is going to have insight into the decisions Congress makes, it should have access to the motions on which Congress acts.

But motions are something of a black hole. Many of them can be found in the Congressional Record, but it takes a human who understands legislative procedure and who is willing to read the Congressional Record to find them. That is not modern transparency.

Motions can be articulated as data. There are distinct types of motions. Congress can publish which meeting a motion occurs in, when the motion occurs, what the proposition is, what the object of the motion is, and so on. Along with decisions, motions are key elements of the legislative process. They can and should be published as data.

Decisions and Votes: B+

When a motion is pending, a body such as the House, the Senate, or a committee will make a decision on it, only sometimes using votes. These decisions are crucial moments in the legislative process, which should be published as data. Like motions, many decisions are not yet published usefully. Decisions made without a vote in the House or Senate are published in text form as part of the Congressional Record, but they are not published as data, so they remain opaque to the Internet. Many, many decisions come in the form of voice votes, unanimous consents, and so on.

Voting puts members of Congress on record about where they stand. And happily, vote information is in pretty good shape. Each chamber publishes data about votes, meaning authority is well handled. Vote data are available and timely.

Both the House³⁰ and Senate³¹ produce vote information. The latter also publishes roll call tables in XML, which is useful for computer-aided oversight. Overall, voting data are pretty well handled. But the omission of voice votes and unanimous consents drags the grade down and will drag it down further as the quality of data publication in other areas rises.

Communications (Inter- and Intra-Branch): F

The Constitution requires each house of Congress to “keep a Journal of its Proceedings, and from time to time publish the same.” The basic steps in the legislative process (discussed elsewhere) go into the journals of the House and Senate, along with communications among governmental bodies.

These messages, sent among the House, Senate, and Executive Branch, are essential parts of the legislative process, but they do not see publication. Putting these communications online—including unique identifiers, the sending and receiving body, any meeting that produced the communication, the text of the communication, and key subjects such as bills—would complete the picture that is available to the public.

Publication Practices for Transparent Government: Budgeting, Appropriations, and Spending

Agencies: D-

Federal agencies are the “agents” of Congress and the president. They carry out federal policy and spending decisions. Accordingly, one of the building blocks of data about spending is going to be a definitive list of the organizational units that do the spending.

Is there such a list? Yes. It’s Appendix C of OMB Circular A-11, entitled: “Listing of OMB Agency/Bureau and Treasury Codes.” This is a poorly organized PDF document that is found on the Office of Management and Budget website.³²

Poorly organized PDFs are not good transparency. Believe it or not, there is still no federal government “organization chart” that is published in a way amenable to computer processing.

There are almost certainly sets of distinct identifiers for agencies that both the Treasury department and the Office of Management and Budget use. With modifications, either of these could be published as the executive branch’s definitive list of its agencies. But nobody has done that. Nobody seems yet to have thought of publishing data about the basic units of the executive branch online in a machine-discoverable and machine-readable format.

In our preliminary grading, we gave this category an “incomplete” rather than an F. That was “beyond generous,” according to Becky Sweger of the National Priorities Project. We expect improvement in publication of this data, and the grades will be low until we get it.

Bureaus: D-

The sub-units of agencies are bureaus, and the situation with agencies applies to data about the offices where the work of agencies get divided up. Bureaus have identifiers. It’s just that nobody publishes a list of bureaus, their parent agencies, and other key information for the Internet-connected public to use in coordinating its oversight.

Again, a prior “incomplete” in this area has converted to a D-, saved from being an F only by the fact that there is a list, however poorly organized and published, by the Office of Management and Budget.

Programs: D

It is damning with faint praise to call “programs” the brightest light on the organizational-data Christmas tree. The work of the government is parceled out for actual execution in programs. Like information about their parental units, the agencies and bureaus, data that identifies and distinguishes programs is not comprehensively published.

Some information about programs is available in usable form. The Catalog of Federal Domestic Assistance website (www.cfda.gov) has useful aggregation of some information on programs, but the canonical guide to government programs, along with the bureaus and agencies that run them, does not exist.

Programs will be a little bit heavier a lift than agencies and bureaus—the number of programs exceeds the number of bureaus by something like an order of magnitude, much as the number of bureaus exceeds the number of agencies. And it might be that some programs have more than one agency/ bureau parent. But today’s powerful computers can keep track of these things—they can count pretty high. The government should figure out all the programs it has, keep that list up to date, and publish it for public consumption.

Thanks to the CFDA, data publication about the federal government’s programs gets a D.

Projects: F

Projects are where the rubber hits the road. These are the organizational vehicles the government uses to enter into contracts and create other obligations that deliver on government services. Some project information gets published, but the publication is so bad that we give this area a low grade indeed.

Information about projects can be found. You can search for projects by name on USASpending.gov, and descriptions of projects appear in USASpending/FAADS downloads, (“FAADS” is the Federal Assistance Award Data System), but there is no canonical list of projects that we could find. There should be, and there should have been for a long time now.

The generosity and patience we showed in earlier grading with respect to agencies, budgets, and programs has run out. There’s more than nothing here, but projects, so essential to have complete information about, gets an F.

Budget Documents—Congress: D/White House: B-

The president’s annual budget submission and the congressional budget resolutions are the planning documents that the president and Congress use to map the direction of government spending each year. These documents are published authoritatively, and they are consistently available, which is good. They are sometimes machine-discoverable, but they are not terribly machine-readable.

The appendices to the president’s budget are published in XML format, which vastly reduces the time it takes to work with the data in them. That’s really good. But the congressional budget resolutions—when they exist—have no similar organization, and there is low correspondence between the budget resolutions that Congress puts out and the budget the president puts out. You would think that a person—or better yet, a computer—should be able to lay these documents side by side for comparison, but nobody can.

For its use of XML, the White House gets a B-. Congress gets a D.

Budget Authority: F

“Budget authority” is a term of art for what probably should be called “spending authority.” It’s the power to spend money, created when Congress and the president pass a law containing such authority.

Proposed budget authority is pretty darn opaque. The bills in Congress that contain budget authority are consistently published online—that’s good—but they don’t highlight budget authority in machine-readable ways. No computer can figure out how much budget authority is out there in pending legislation.

Existing budget authority is pretty well documented in the Treasury Department’s FAST book (Federal Account Symbols and Titles). This handy resource lists Treasury accounts and the statutes and laws that provide their budget authority. The FAST book is not terrible, but the only form we’ve found it in is PDF. PDF is terrible. And nobody among our graders uses the FAST book.

Congress can do a lot better, by highlighting budget authority in bills in a machine-readable way. The administration can do much, much better than publishing the obscure FAST book in PDF.

Ideally, there would be a nice, neat connection from budget authority right down to every outlay of funds, and back up again from every outlay to its budget authority. These connections, published online in useful ways, would allow public oversight to blossom. But the seeds have yet to be planted.

Warrants, Apportionments, and Allocations: F

After Congress and the president create budget authority, that authority gets divvied up to different agencies, bureaus, programs, and projects. How well documented are these processes? Not well.

An appropriation warrant is an assignment of funds by the Treasury to a treasury account to serve a particular budget authority. It’s the indication that there is money in an account for an agency to obligate and then spend. “OMB has a web portal that agencies used to send apportionment requests,” notes the National Priorities Project’s Becky Sweger, “so the apportionment data are out there.”

Where is this warrant data? We can’t find it. Given Treasury’s thoroughness, it probably exists, but it’s just not out there for public consumption.

An apportionment is an instruction from the Office of Management and Budget to an agency about how much it may spend from a Treasury account in service of given budget authority in a given period of time.

We haven’t seen any data about this, and we’re not sure that there is any. There should be. And we should get to see it.

An allocation is a similar division of budget authority by an agency into programs or projects. We don’t see any data on this either. And we should.

These essential elements of government spending should be published for all to see. They are not published, garnering the executive branch an F.

Obligations: B-

Obligations are the commitments to spend money into which government agencies enter. Things like contracts to buy pens, hiring of people to write with those pens, and much, much more.

USASpending.gov has quickly become the authoritative source for this information, but it is not the entire view of spending, and the data is “dirty”: inconsistent and unreliable. The use of proprietary DUNS numbers—the Data Universal Numbering System of the firm Dun & Bradstreet—also weakens the availability of obligation data.

There is some good data about obligations, but it is not clean, complete, and well documented. The ideal is to have one source of obligation data that includes every agency, bureau, program, and project. With a decent amount of data out there, though, useful for experts, this category gets a B-.

Parties: F

When the government spends taxpayer dollars, to what parties is it sending the money?

Right now, reporting on parties is dominated by the DUNS number. It provides a unique identifier for each business entity and was developed by Dun & Bradstreet in the 1960s. It’s very nice to have a distinct identifier for every entity doing business with the government, but it is not very nice to have the numbering system be a proprietary one.

“Parties” would grade well in terms of machine-readability, which is one of the most important measures of transparency, but because it scores so low on availability, its machine-readability is kind of moot. Until the government moves to an open identifier system for recipients of funds, it will get weak grades on publication of this essential data.

Outlays: C-

For a lot of folks, the big kahuna is knowing where the money goes: outlays. An outlay—literally, the laying out of funds—satisfies an obligation. It’s the movement of money from the U.S. Treasury to the outside world.

Outlay numbers are fairly well reported after the fact and in the aggregate. All one has to do is look at the appendices to the president’s budget to see how much money has been spent in the past.

But outlay data can be much, much more detailed and timely than that. Each outlay goes to a particular party. Each outlay is done on a particular project or program at the behest of a particular bureau and agency. And each outlay occurs because of a particular budget authority. Right now these details about outlays are nowhere to be found.

“Surely the act of cutting a check doesn’t sever all relationship between that amount of money and its corresponding obligation/ project/program,” writes a frustrated Becky Sweger from the National Priorities Project. “Surely these relationships are intact somewhere and can be published.”

Plenty of people inside the government who are familiar with the movement of taxpayer money will be inclined to say, “it’s more complicated than that,” and it is! But it’s going to have to get quite a bit less complicated before these processes can be called transparent.

The time to de-complicate outlays is now. It’s a feat of generosity to give this area a C-. That’s simply because there is an authoritative source for aggregate past outlay data. As the grades in other areas come up, outlay data that stays the same could go down. Way down.

Conclusion

Many of the entities discussed here are low-hanging fruit if Congress and the administration want to advance transparency and their transparency grades. Authoritative, complete, and well-published lists of House and Senate membership, committees, and subcommittees are easy to produce and maintain, and much of the work has already been done.

The same is true of agencies and bureaus, at least on the executive branch side. Presidential leadership could produce an authoritative list of programs and projects within months. Establishing authoritative identifiers for these basic units of government is like creating a language, a simple but important language computers can use to assist Americans in their oversight of the federal government.

The more difficult tasks—amendments to legislation, for example, and discretely identified budget authorities—will take some work. But such work can produce massive strides forward in accountable, efficient, responsive, and—in the libertarian vision—smaller government.

Appendix A

Appendix B

Notes

1. An illustration of the focus on the topic, Politifact.com’s “Obameter” lists 35 Obama campaign promises from 2008 related to transparency, http://www.politifact.com/truth-o-meter/promises/obameter/subjects/transparency/.

2. Macon Phillips, “Change Has Come to Whitehouse.gov,” The White House Blog, January 20, 2009 (12:01 p.m. EDT), http://www.whitehouse.gov/blog/change_has_come_to_whitehouse-gov.

3. Barack Obama, “Transparency and Open Government,” Presidential Memorandum (January 21, 2012), http://www.whitehouse.gov/the-press-office/transparency-and-open-government.

4. Eric Holder, “Memorandum for Heads of Executive Departments and Agencies: The Freedom of Information Act (FOIA),” Office of the Attorney General (March 19 2009), http://www.justice.gov/ag/foia-memo-march2009.pdf.

5. Jim Snyder and Danielle Ivory, “Obama Cabinet Flunks Disclosure Test with 19 in 20 Ignoring Law,” Bloomberg.com, September 28, 2012, http://www.bloomberg.com/news/2012-09-28/obama-cabinet-flunks-disclosure-test-with-19-in-20-ignoring-law.html.

6. Jim Harper, “Sunlight before Signing: Measuring a Campaign Promise,” Cato@Liberty (blog), September 6, 2012, http://www.cato-at-liberty.org/sunlight-before-signing-measuring-a-campaign-promise/.

7. Jesse Lee, “Transparency and Open Government,” May 21, 2009, http://www.whitehouse.gov/blog/2009/05/21/transparency-and-open-government.

8. Peter R. Orszag, “Memorandum for the Heads of Executive Departments and Agencies, Subject: Open Government Directive,” M 10-06, December 8, 2009, http://whitehouse.gov/open/documents/open-government-directive.

9. “Negotiate Health Care Reform in Public Sessions Televised on C-SPAN,” Politifact.com, http://www.politifact.com/truth-o-meter/promises/obameter/promise/517/health-care-reform-public-sessions-C-SPAN/.

10. Alon Peled, “When Transparency and Collaboration Collide: The USA Open Data Program,” Journal of the American Society for Information Science and Technology 62, no. 11 (November 2011): 2085.

11. Harlan Yu and David G. Robinson, “The New Ambiguity of ‘Open Government,’” UCLA Law Review 59, no. 6 (August 2012): 178.

12. Jim Harper, “Grading Agencies’ High-Value Data Sets,” Cato@Liberty (blog) February 5, 2010, http://www.cato-at-liberty.org/grading-agencies-high-value-data-sets/.

13. Jim Harper, “Is Government Transparency Headed for a Detour?” Cato@Liberty (blog), January 15, 2010, http://www.cato-at-liberty.org/is-government-transparency-headed-for-a-detour/.

14. Yu and Robinson, p. 182.

15. Open Government Partnership, http://www.opengovpartnership.org/.

16. Office of the Press Secretary, The White House, “Fact Sheet: The Open Government Partnership,” September 20, 2011, http://www.whitehouse.gov/the-press-office/2011/09/20/fact-sheet-open-government-partnership.

17. “John Boehner Introduces the House GOP Congressional Transparency Initiative,” http://www.youtube.com/watch?v=hDr70qRv_9k.

18. Library of Congress, “About THOMAS,” http://thomas.loc.gov/home/abt_thom.html.

19. The text of the House rules package is available at http://www.rules.house.gov/News/PRArticle.aspx?NewsID=33.

20. Committee on House Administration, Legislative Data and Transparency Conference, http://cha.house.gov/about/contact-us/legislative-data-conference.

21. Andrew Weber, “Introducing Congress gov!” Custodia Legis: Law Librarians of Congress (blog) September 19, 2012, http://blogs.loc.gov/law/2012/09/introducing-congress-gov/.

22. See Jim Harper, “Congress on Transparency: ‘Needs Improvement,’” Cato@Liberty (blog), September 23, 2011, http://www.cato-at-liberty.org/congress-on-transparency-needs-improvement/; and Jim Harper, “Government Spending Transparency: ‘Needs Improvement’ Is Understatement,” Cato@Liberty (blog), December 14, 2011, http://www.cato-at-liberty.org/government-spending-transparency-%E2%80%98needs-improvement%E2%80%99-is-understatement/.

23. Jim Harper, “Publication Practices for Transparent Government,” Cato Institute Briefing Paper no. 121, September 23, 2011, http://www.cato.org/publications/briefing-paper/publication-practices-transparent-government.

24. See Jim Harper, “Congress on Transparency: ‘Needs Improvement’”; and Jim Harper, “Government Spending Transparency: ‘Needs Improvement’ Is Understatement.”

25. U.S. Senate, “Senators of the 112th Congress,” http://www.senate.gov/general/contact_information/senators_cfm.cfm.

26. U.S. Senate, “Committees,” http://www.senate.gov/pagelayout/committees/d_three_sections_with_teasers/committees_home.htm.

27. U.S. House of Representatives, “Document Naming Conventions,” http://cha.house.gov/sites/republicans.cha.house.gov/files/documents/committee_docs/CommitteeRepository-NamingConventions-v1-2-1.pdf.

28. U.S. Senate, “Daily Digest Committee Meetings/Hearings Schedule,” http://www.senate.gov/pagelayout/committees/b_three_sections_with_teasers/committee_hearings.htm.

29. U.S. Government Printing Office, “Congressional Reports,” http://www.gpo.gov/fdsys/browse/collection.action?collectionCode=CRPT.

30. U.S. House of Representatives, Office of the Clerk, “Legislation and Reports,” http://clerk.house.gov/legislative/legvotes.aspx.

31. U.S. Senate, “Recent Votes,” http://www.senate.gov/pagelayout/legislative/a_three_sections_with_teasers/votes.htm.

32. U.S. Office of Management and Budget, “Circular A-11,” http://www.whitehouse.gov/omb/circulars_a11_current_year_a11_toc.