IBM Technical Support QA Leaderboard

×
×

What is TechQA?

Welcome to IBM TechQA, a domain-adaptation question answering leaderboard for the technical support domain.

The TechQA leaderboard highlights two real-world issues from the automated customer support domain. First, it contains actual questions posed by users on a technical forum. Second, it has a real-world size 600 training, 310 dev, and 490 test question/answer pairs.

The TechQA dataset purpose is to stimulate research in domain adaptation rather than as a resource to build QA systems from scratch.

Resources

The techQA dataset consists of a collection of 800k technical documents, called Technotes, that address specific technical questions, and of annotated questions with answers in the collection.

Questions are divided into a training set, containing 450 answerable and 150 non-answerable questions, and a development set, containing 160 answerable and 150 unanswerable questions.

To access the dataset, please register for the leaderboard.

Examples of questions and expected system answers.

The TechQA dataset contains questions consisting of a Title and a Body. Each question is associated with 50 Technotes, technical documents that address specific topics.
Questions that have an answer in one of the 50 documents are considered answerable, while the remaining ones are considered unanswerable.
Each answerable question is paired with an answer consisting of a technote id and the start and end offset of the answer.

Technotes on the web look like this. The technotes released with TechQA have been preprocessed: the navigation controls and the infobox have been removed: the same technote now looks like this.
Each technote contains the preprocessed html page as well as the detagged text
Metadata is available with each of the technotes released with TechQA that capture some of the information removed by the preprocessing.

Structure of the examples

TITLE: This will contain the title of the question.
BODY: This will contain the body of the question.
REMARKS: A brief remark about question and answer,
ANSWER: The text to the right of ANSWER is the answer selected by the annotators.
To see the answer in context, click
Technote
This shows the full technote text with the answer in red.

Examples

TITLE: Does Linux KVM monitoring agent support CANDLEDATA function?
BODY: Does Linux KVM monitoring agent in ITM support CANDLEDATA function?
REMARKS:This is a yes/no question; note that the answer must contain the supporting evidence.
ANSWER: The CANDLEDATA mechanism is available with ITM 6.30.05.00.


TITLE: Does DataPower support SHA-2?
BODY: Is DataPower able to support SHA-2
REMARKS:  This is a yes/no question, note that the answer must contain the supporting evidence.
ANSWER: Yes, if a Certificate Authority(CA) signs a certificate using SHA2(SHA-256), IBM Websphere DataPower SOA Appliance can validate CA's signature on the certificate while building trust chain from the DataPower ValCred.


TITLE:What exactly is "wpcollector" in WebSphere Portal Server?
BODY:I've been told to get wpcollector output? What exactly is this?
REMARKS:This is a "non-factoid" informational question related to a program. Note also that the title indicates that "this" in the body refers to "wpcollector".
ANSWER:Wpcollector is a command line tool that automates the collection of portal logs and configuration files. Using automated log collection early in the Case life cycle can greatly reduce the number of doc requests that are made by Support.


TITLE: How do I identify Indexing errors in Atlas database?
BODY: How do I identify Indexing errors in Atlas database?
REMARKS:   This is an "how-to" question, explicitly related to a diagnostic task. Note that title and body are identical.
ANSWER: 1. Login to Atlas database as PSSAPL user
2. Run select * from ctx_user_index_errors;
3. Copy the value in the ERR_TEXTKEY Field
4. Run select * from documentlibrary where rowid='<value from Step 3> ';
5. This will list the document which has the indexing error


TITLE: MQSeries Events correlation
BODY: How are MQSeries Events cleared?
REMARKS:   This is a "non-factoid" informational question.
ANSWER:The MQSeries Events workspace displays data from the Current Events attribute group. Since this data is stored in memory, stopping and restarting the MQ Monitoring agent will clear all the data in the Current Events table.


TITLE: How to find if Table was Created as NOT LOGGED INITIALLY?
BODY: How to find if Table was Created as NOT LOGGED INITIALLY?
REMARKS:   This is a "non-factoid" informational question. Note that the question is very specific. Note also how the title and the body of the question are identical.
ANSWER:We can use either of the following methods to Check if Table was created as 'Not Logged Initially'

1)Using db2cat,
db2cat -db <dbname> -n <TableName> -s <Schema> -t | grep "Table not logged"
Which can confirm if given table was created as NLI or not.
Sample Output
Connecting to the Database.
Connect to Database Successful.
Table not logged : 0
Table not Logged showing '0' represent Table was not created as NLI, if it returns '1' then that is table was created as NLI.

2)Using db2look
Using db2look command to get DDL information on the table which would show you if
the table was created as "NOT LOGGED INITIALLY"


TITLE User environment variables no longer getting picked up after upgrade to 4.1.1.1 or 4.1.1.2?
BODY Have you found that after upgrade to Streams 4.1.1.1 or 4.1.1.2, that environment variables set in your .bashrc are no longer being set?
For example ODBCINI is not set for the database toolkit and you get
An SQL operation failed. The SQL state is 08003, the SQL code is 0 and the SQL message is [unixODBC][Driver Manager]Connnection does not exist.
REMARKS:  The user asks implicitly for the procedure to solve a problem.
ANSWER:To work around the issue, set environment variables that are needed by the application directly in the instance with: *
* streamtool setproperty
* -d <domain> -i <instance>
--application-ev <VARIABLE NAME>=<VARIABLE VALUE>


TITLE Unable to locate the More tab of Document class - Property definition tab - property template configuration to update Is Required setting.
BODY Unable to locate the More tab of Document class - Property definition tab - property template configuration to update Is Required setting. Environment is running on Content Engine 5.2.0.3.
REMARKS:   This question asks implicitly for the procedure to solve a problem.
ANSWER:Follow the below steps to update the property definition properties on ACCE:
1. Launch ACCE
2. Expand the object stores folder and click the object store to open the Object store tab
3. Expand the Data Design folder and Classes folder. Click the document class to open Document Class tab
4. From Properties tab, open Property Definitions drop down and select the property definition you want to modify
5. ACCE will open that property definition in a new Properties tab
6. Modify the property definition as required
7. Go back to the Class Definition tab and click Save


TITLE Why are we not able to create new pages using the Manage Pages Portlet?
BODY We are using Portal 8.5 on Windows,
From WebSphere Portal Administration console. we go to ‘Manage Pages’- > Select ‘Content root’ item from table -> ‘New Page’ button, after entering the new page details and hitting OK we receive following error:
EJPAS0017E: Unable to create PageName
The error in SystemOut.log shows:
   DefaultTransa E Message: Could not save object in repository.,
   Cause: javax.jcr.RepositoryException: RT0002E: Error while calling a function createItems of PLS data manager
   com.ibm.workplace.wcm.services.repository.RepositoryException:
   Message: Could not save object in repository.,
   Cause: javax.jcr.RepositoryException: RT0002E: Error while calling a function createItems of PLS data manager
   ...
   Caused by: com.ibm.icm.da.DBAccessException:
   java.sql.SQLFeatureNotSupportedException: DSRA1300E: Feature is not implemented: PreparedStatement.setBinaryStream
   at com.ibm.icm.da.portable.data.WideTableData.changeItems(WideTableData.java:857)
REMARKS:   This question asks for the root cause of a problem. Not how the question is in the Title, while the Body contains a description of the error.
ANSWER:The JDBC driver does not meet WebSphere Portal v8's prerequisites.


TITLE Request fails with "non idempotent request method - RFC 2616 sec 10.3. "
BODY When a request is routed through DataPower and a redirect is done on a POST, the redirect will fail in DataPower and we can expect the following message to be logged in the error message: "non idempotent request method - RFC 2616 sec 10.3." How do I solve the problem?
REMARKS:   This question asks explicitly how to solve a problem; the document contains the high-level answer, in red, and points to details on the actual procedure.
ANSWER:To resolve the issue, the IBM WebSphere DataPower SOA appliance web application firewall can be configured to handle these "302 Redirects".


Leaderboard

Rank Model F1 HA_F1@1 HA_F1@5 BEST_F1
Rank Model F1 HA_F1@1 HA_F1@5 BEST_F1

Registration

Please use the Sign Up link to register for TechQA.
Registration is a two-step procedure: after providing your contact e-mail you will receive a message with a link to the page where you will be able to complete the registration process. Once you register, you will be able to access the TechQA dataset, to submit entries for the leaderboard, and monitor and review your submissions.

Login

Please log in with the credentials you used for registering to access the full functionality of the web site.

Login

Forgot Password?

Submission Lifecycle

  • Create a new submission with the CREATE SUBMISSION. Your submission initial State is pulling
  • We retrieve the image from the repository. Upon success the State changes to queued_test followed by testing; upon failure the State changes to error and no further action is taken.
  • We run your image against the validation set. Upon successful completion, the State changes to tested and the computed metrics are available in the Scores column.
  • If you are satisfied with the Scores, issue the Run Eval. Your submission will be queued for evaluation and its state changed to queued|running.
  • Your image will run in a machine with 120 Gb of memory and 2 V100 GPUs. You will have exclusive use of the machine for up to 24 hours. The container will not be able to access the network. If the run does not finish in 24 hours, it is stopped and placed in aborted state.
  • Once the evaluation completes, the Scores are updated, the state changes to done and the scores are published on leaderboard.

How to manage your submissions

Cancel
Will terminate the submission, removing it from the queue or stopping its execution.
Command to anonymize
If you elect to anonymize your submission, the affiliation will be changed to Anonymous and the paper link will be hidden
Command to de-anonymize
If you elect to de-anonymize your submission, the affiliation and paper link will be disclosed on the leaderboard

Submission Status Codes

System StateState Description
pullingfetch of the image starting initial state
queued_testimage fetched, test put on queue
testing test started
testedtest finished, test scores available
queuedafter owner clicked 'Run Eval'
runningthe execution started
noimageerror, failed to fetch image, final state
cancellingafter user clicked 'Cancel',
cancelledcncellation completed final state
abortedtime allowed exceeded, final state
errorunrecoverable error during execution, final state
donesucces, final scores available and published onleaderboard, final state
no_scoreserror, the execution finished without errors but didn't produces scores, final state

My Submissions

System State Submitted Completed Action Submission Id
System State Submitted Completed Action Submission Id

Submission Form Instructions

To create a new submission please fill the fields as follows:

  • System Name   Name the system with a name you can recognize even when the submission is anonymized on the leaderboard.
  • Paper URL   Optionally provide the url of a paper describing your system. Leave blank otherwise.
  • Full Docker Image Name   Upload your image to a docker registry whose credentials you can share with us (ensure that only images you want us to fetch are accessible with the credentials). Fill this field with the full image name.
  • Docker Registry Server   Provide the Registry server hostname - leave blank if using Docker Hub.
  • Registry User   Fill with the username we can use to log into the registry server.
  • Registry Access Key   Fill with the access key we can use to log into the registry server.
  • Anonymous or Public   If you want your affiliation and paper link to appear on the leaderboard, select Public. If you want it to appear as Anonymous (e.g., for paper review purpose), select Anonymous. You will be able to change this selection at a later time.
We will discard the repository information once the image is successfully retrieved.

Example:

  • Full Docker Image Name: local.artifactory.ww-devops.com/tech-qa-system1:submission1
  • Docker Registry Server Name: local.artifactory.ww-devops.com
  • Docker Registry User Name:me@myemail.com
  • Docker Registry Access Key *XXXXXXXXX*

New Submission


This page contains a description of the dataset, the license, and instructions on how to prepare an image for sumbission
By submitting your image you certify that no data from the ibm.com/developwerworks or developer.ibm.com sites, other than what is provided with this dataset, were used to build any of the components of your system, including models, rules, or dictionaries.

The dataset

This is a description of the content of the various directories and of the data format The training_and_dev directory contains the following files:
training_Q_A.json the training set
dev_Q_A.json the development set
training_dev_technotes.json   documents referenced in the training and development set

The training_Q_A.json and dev_Q_A.json files contain lists of maps. Each maps represents a labeled instance

  training_Q_A.json   contains 600 labeled instances
dev_Q_A.json contains 310 labeled instances

Each instance consists of the following fields:

System inputs
  QUESTION_ID: the unique id of the instance
QUESTION_TITLE: the title of the question - a summary, introduction, and occasionally a full question
QUESTION_BODY:   typically a longer span of text than the QUESTION_TITLE, contains the actual question asked by the user. Occasionally it is identical to the title text
DOC_IDS: a list of strings: each is a unique document id, found in training_dev_technotes.json systems are asked to find an answer to the question among these documents, if available
System predictions
ANSWERABLE: 'Y' if human annotators found an answer in one of the documents in DOC_IDS
'N' otherwise
DOCUMENT: if ANSWERABLE is 'Y': the id of the document containing the answer
if ANSWERABLE is 'N': the value is '-'
START_OFFSET:   if ANSWERABLE is 'Y': zero-based character offset of the first character of the answer in the 'text' field of the document having 'id' == DOCUMENT
if ANSWERABLE is 'N': the value is '-'
END_OFFSET: if ANSWERABLE is 'Y': zero-based character offset of the end of the answer in the 'text' field of the document having 'id' == DOCUMENT, using PYTHON / C conventions
if ANSWERABLE is 'N': the value is '-'
ANSWER: if ANSWERABLE is 'Y': the string document['text'][START_OFFSET:END_OFFSET] where document is the document having 'id'=DOCUMENT
if ANSWERABLE is 'N': the value is '-'

The training_dev_technotes.json file contains the subset of the Technotes collection that is referenced in the training_Q_A.json and dev_Q_A.json files.
Each document contains the following fields

  'id' unique document id
'content' html version of the page
'title' title of the page, detagged
'text' automatically detagged body of the page
'metadata' map with the following fields:
'sourceDocumentId'   same as 'id' in most cases
'date' date of last modification
'productName' name of the product for which the technote document was written
'productId' id of the product, could be useful for cross-referencing or disambiguating products
'canonicalUrl' canonical URL of the page - does not necessarily correspond to the actual URL of the page

The validation folder contains files that will be used to verify that a submitted system is running correctly. Specifically, when a submission is received, the system will be run against the validation_questions, and the output scored against validation_reference.json.

The scores produced by the evaluation.py script will be returned to the submitter who can check them against a local run. If the submitter decides that the scores are correct, then the submitter can trigger the official evaluation

The validation folder contains the following files:

  validation_questions.json input file for the system, containing the questions arranged as an array of maps
Each maps contains the QUESTION_ID, QUESTION_TITLE, QUESTION_TEXT, DOC_IDS fields described above.
validation_reference.json ground truth against which the system output is evaluatated. The file has the same format as the training/dev sets described above.
validation_technotes.json The collection of technotes mentioned in validation_questions.json. Has the same format as training_dev_technotes.json
validation_example_output.json Example output of a system. The scorer expects the system output to be in this format. See 'Output Format' below.
This specific file is not the actual output of a system but was randomly generated using the ground truth.

Technote Corpus

The technote_corpus contains full_technote_collection.txt.bz2, the collection of the IBM technotes available on the web as of April 4, 2019 - 801998 technotes in total.

Technotes are IBM documents maintained by IBM support personnel that contain information about common questions asked by customers, including product upgrade information and solutions to problems. They cover a broad variety of products, including many that are not produced by IBM.

We believe this to be a useful resource, for example for fine-tuning language models for the technical support domain.

Data format

The full_technote_collection.txt.bz2 file contains one json array per line. Each entry of the array is a map with the fields described above in the training_dev_technotes.json section. The file can be read following the reader.py example code, which produces a dictionary indexed by the document 'id'.


License

The TechQA Dataset is released under the Community Data License Agreement - Permissive - Version 1.0. Please see the CDLA-Permissive-v1.0.pdf file in the top-level directory of the dataset.

Preparing and Submitting and Image

How to prepare your image

The system has to be packaged in a ready-to-run Docker image that will read two input files and write the system output in a third file.

The two input files will be mounted read-only inside the image:

 /var/spool/TechQA/input/questions.json
 /var/spool/TechQA/input/documents.json
Please note that the parent directories are also read-only.

question.json has the same format as validation_question.json and documents.json has the same input format as validation_technotes.json, both described above.

The output will be written by your system to /var/spool/TechQA/output, which will be mounted as writeable directory. We expect your system to write the results in /var/spool/TechQA/output/sys_output.py. All other files produced by your system will be deleted upon completion of the run.

NOTE: Your image must be self-contained because it will run without network access: ready-to-run means that all the resources you need must be in the image.

NOTE: Your image must be able to run without extra command-like arguments.

How to submit the image

Detailed instructions on how to submit the image are in the submission page, reacheable from "My Submissions".
Note that the submission image must be availabe either in a public or a private Docker registry.

You will need the following information to submit an image:

  • System Name   Name the system with a name you can recognize even when the submission is anonymized on the leaderboard.
  • Paper URL   Optionally provide the url of a paper describing your system. Leave blank otherwise.
  • Full Docker Image Name   Upload your image to a docker registry whose credentials you can share with us (ensure that only images you want us to fetch are accessible with the credentials). Fill this field with the full image name.
  • Docker Registry Server   Provide the Registry server hostname - leave blank if using Docker Hub.
  • Registry User   Fill with the username we can use to log into the registry server.
  • Registry Access Key   Fill with the access key we can use to log into the registry server.
  • Anonymous or Public   If you want your affiliation and paper link to appear on the leaderboard, select Public. If you want it to appear as Anonymous (e.g., for paper review purpose), select Anonymous. You will be able to change this selection at a later time.

Output Format

The system output must be in json format. It must consist of a dictionary with two keys, threshold and predictions.
The value of the threshold key is a (floating point) number. The value of the predictions key is a dictionary, indexed by the QUESTION_ID of the questions in the input file.
The value associated with each QUESTION_ID is a list containing 5 dictionaries. The first dictionary in the list contains the top prediction produced by the system, and the others contain the next 4 predictions. Each prediction must be a dictionary with the following keys:

  doc_idThe id of the document from which the prediction was extracted
  start_offset The zero-based character offset of the first character of the answer, from the text field of the document
  end_offset The zero-based character offset of the character after the last character of the answer, from the text field of the document
  score a floating point number, compared to the threshold: if larger or equal, the scorer assumes that the document contains an answer, if smaller the scorer assumes that the system does not believe the document contains an answer

Example

{
"threshold": -0.5495,
"predictions": {
"VAL_Q000": [
{ "doc_id": "swg24042191", "start_offset": 955, "end_offset": 1057, "score": -0.0208 },
{ "doc_id": "swg21961858", "start_offset": 1835, "end_offset": 903, "score": -1.7697 },
{ "doc_id": "swg21961858", "start_offset": 1160, "end_offset": 47, "score": 0.2268 },
{ "doc_id": "swg21984451", "start_offset": 1473, "end_offset": 1346, "score": 0.7463 },
{ "doc_id": "swg21980056", "start_offset": 1809, "end_offset": 360, "score": -1.0812 }
], .
"VAL_Q001": [ ...
}
}

Contact us

Do you have any questions? Please do not hesitate to contact us directly. Our team will come back to you within a matter of hours to help you.