Error processing job

error processing job

Describe the bug I am trying to run a Processing job and the job fails part way with this error message: Failed. The system error 61:05 Error Processing Job error indicates there is a communication problem and the HP Designjet 800 or 500 series printer was not able to. When these errors occur the process is not killed. Instead these jobs just repeatedly get re-attempted and never removed. I think because the.

Error processing job - right! think

Error Processing Representative job

Primary City/State:

Phoenix, Arizona

Department Name:

Error Processing-Ref Lab

Work Shift:

Day

Job Category:

Revenue Cycle

Flexible start time anywhere from 6am-8am depending on schedule.

POSITION SUMMARY

The Error Processing Representative is primarily responsible for reviewing accessions that fall into the Error Processing workflow categories and for obtaining any, and all missing billing information that is required to finalize claims in a timely manner. The Error Processing Representative is responsible for managing a high volume of work on a daily basis. To accomplish our goals, the following essential functions will be used and all employees are responsible for supporting the company’s goals, mission, and values; following all company policies and procedures; providing quality customer service; and following safety standards in performing all aspects of the functions below.

CORE FUNCTIONS

  1. Researches information received from the client, from websites, and/or from previous billed claims to obtain information that is missing or invalid.

  2. Completes daily assignments and projects as instructed by the Supervisor, Assistant Supervisor, or QC/Trainer.

  3. Reconciles returned requests and accurately enters data into the system in a timely manner.

  4. Responds to phone calls from clients to resolve billing issues as well as makes outgoing calls as needed.

  5. Has a working knowledge of internal client’s duties and responsibilities.

  6. Assists other billing sub-departments as needed.

MINIMUM QUALIFICATIONS

  • High School Diploma/GED.

  • Three (3) years’ experience in Medical Billing, Medical Office, or related field.

  • Ability to work independently and as a team member.

  • Ability to work with high volumes of data.

  • Ability to meet deadlines.

PREFERRED QUALIFICATIONS

  • Previous experience in Medical Billing and Coding.

  • Medical coding and computer knowledge.

  • Additional related education and/or experience.

Banner Health is one of the largest, nonprofit health care systems in the country and the leading nonprofit provider of hospital services in all the communities we serve. Throughout our network of hospitals, primary care health centers, research centers, labs, physician practices and more, our skilled and compassionate professionals use the latest technology to make health care easier, so life can be better. The many locations, career opportunities, and benefits offered at Banner Health help to make the Banner Journey unique and fulfilling for every employee.

EOE/Female/Minority/Disability/Veterans Banner Health supports a drug-free work environment. https://www.bannerhealth.com/careers/eeo

Application currently closed
This job was posted on Sat Apr 16 2022 and expired on Fri Apr 29 2022.

Job server stops processing due to error, "Processing is disabled (queues globally switched off)" (4247672)

产品:
Identity Manager
8.1.5, 8.1.4, 8.1.3, 8.0.5, 8.1.2, 8.1.1, 8.1, 8.0.4, 8.0.3
Identity Manager On Demand
Hosted
主题:
Troubleshooting
文章历史记录:
创建时间: 8/22/2022
上次更新时间: 8/24/2022

Your Request will be reviewed by our technical reviewer team and, if approved, will be added as a Topic in our Knowledgebase.

您可以在附属支持站点上查找适用于戴尔*产品*的在线支持帮助。单击“继续”,转至适用于*产品*的正确支持内容和帮助。

© 2022 One Identity LLC. ALL RIGHTS RESERVED. Feedback 使用条款 隐私

One Identity门户不再支持IE 8、9和10,建议将您的浏览器升级到最新版本的Internet Explorer或Chrome。

要升级到IE 11,请单击此处

要升级到Chrome,请单击此处

如果继续使用IE 8、9和10,您将无法充分利用我们所有出色的自助服务功能。

Processing error workflow

This topic provides information on working with errors that may occur during processing jobs.

This page contains the following information:

Required security permissions

The following security permissions are required to perform actions on File Errors:

Object SecurityTab Visibility
  • Discovered File - View, Edit

  • Download and Replace files with processing errors

The following security permissions are required to perform actions on Job Errors:

Object SecurityTab Visibility
  • Job Error - View, Edit

  • Processing Error - View, Edit

For more information on permissions, see Workspace permissions.

Processing errors tabs

The Files and Job Errors tabs allow you to easily locate issues that may have occurred in any processing sets. These errors are notified to you through the processing set page upon job completion.

Files tab

The Files tab contains all error information associated to specific files that have occurred during the discovery, publish, and deletion phases of processing. The Current Errored Files and All Errored Files views are tailored to error workflow by only containing information pertaining to the errors that have occurred.

  • The Current Errored Files view displays all outstanding errors from processing jobs. This is your primary location for workflows like error retry, ignore, and file replacement.

  • The All Errored Files view is primarily utilized for historical reporting of errors from processing sets. This view displays any file that has encountered an error, regardless of whether it was resolved or not. It is a good reference to export an error report out of Relativity for a given collection or set of custodians.

For more information on these views, see Files tab.

Job Errors tab

The Job Errors tab contains all errors that occurred on processing sets run in your workspace. These errors are usually not associated with any specific files within a processing set, but rather to the entire set itself.

Job Error views

The Current Job Errors view in the Job errors tab displays all unresolved job errors while the All Job Errors view displays any job error that has occurred throughout the lifecycle of the matter. Both views contain the following fields:

  • Error Identifier - the unique identifier of the error as it occurs in the database. When you click this message, you are taken to the error details layout, where you can view the stack trace and other information. Note that for Unresolvable errors, the console is disabled because you can't take any actions on that error from inside Relativity. For more information, see Processing error workflow.
  • Error Status - the status of the error. This is most likely Unresolvable.
  • Message - the cause and nature of the error. For example, "Error occurred while trying to overlay deduplication details. Please resolve publish error or republish documents from data source below. DataSource Artifact Id: 1695700".
  • Custodian - the custodian associated with the data source containing the file on which the error occurred.
  • Processing Set - the name of the processing set in which the error occurred.
  • Data Source - the data source containing the file on which the error occurred.
  • Error Created On - the date and time at which the error occurred during the processing job.
  • Republish Required - the error must be retried in order to be successfully published.
  • Notes - any manually added notes associated with the error.

Note: Errors occurring during inventory are always represented as Job Errors. For more information, see Inventory Errors.

Job Error layout

Clicking on the job error identifier value brings you to the Job Error Details layout.

Note that the Error Actions console is disabled for unresolvable job errors, since you can't retry or ignore job errors the way you can document errors.

Job error details

To see the job error's stack trace, click on the Advanced tab of the error details layout and view the value in the Stack Trace field.

Job error stack trace

Job-level error workflow

Useful error field information

The following sections provide information on error-specific fields and views that you can use in your processing workflow.

Combined error fields

The Files tab displays a single error associated to a file.

This error displays through the Error Message, Error Category, Error Phase, and Error Status fields. However, multiple errors can be associated to a single file at the same time as issues can occur during different phases of Processing. Relativity determines the displayed error based on a set precedence of Processing phases that could potentially block content from being published. The precedence is as follows:

  • Delete – a document was deleted from Relativity, but encountered an issue, potentially affecting recalculation of deduplication.

  • Publish – a document was supposed to be promoted to review but encountered an error and was held back.

  • Discover –a file may have encountered an issue during expansion and may not have extracted a child record and/or associated metadata.

  • Text Extraction – a file encountered an issue during text extraction and is missing some or all associated text.

Error status information

The Error Status field provides information on where the file is in error remediation.

This is helpful to determine if any further actions are required on a file or to see if an error had ever occurred on a record. When a file has all its errors resolved, the Error Message, Error Category, and Error Phase fields no longer display any content, but the Error Status field keeps a status of Resolved to indicate that it was a record that initially encountered issues but have since been fixed. The statuses of errors are as follows:

  • Not Resolved – The error is still outstanding.

  • Resolving – The error was submitted or in the process of being retried.

  • Resolved – The error was resolved.

  • Ignored – The error was ignored. See File error actions.

Error Category list

The Error Category field provides insight into the nature of the errors that have occurred during processing.

The following table provides a listing of all values on the Error Category field, along with a description of what kinds of issues those values bring back if filtered.

Category nameDescription
Corrupt Container

These errors are exclusive to container files that have encountered corruption when attempting to open and locate files within the container itself. When containers have these errors associated to them, you will not see any extracted loose files. These errors are usually either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.

Corrupt FileThese errors are exclusive to non-container files that have found elements of corruption during Processing. These errors are either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.
Could Not IdentifyRelativity Processing was unable to identify the file during Discovery. This may indicate corruption in the file but was unable to be determined at the time of discovery.
EnvironmentalThese errors are caused by issues in the Relativity Processing environment. These should be retried and resolved when encountered.
File Read / Write FailureThese errors are a subset of Environmental issues specifically caused by file system issues. These should be retried and resolved when encountered.
Missing AttachmentAn attachment from a document or email was not able to be extracted from its file.
Missing File MetadataA file is missing a piece of metadata.
Missing Extracted TextThese errors represent issues that occurred during Text Extraction jobs that have caused a file to be missing some or all associated text. A specific root cause was unable to be readily identified, but they should be retried and resolved where possible.
Partially Corrupted ContainerThese errors are exclusive to container files that have encountered corruption during extraction of specific records. When containers have these errors associated to them, you may see some files extracted, but not all. These errors are usually either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.
Password Protected Container

These errors are exclusive to container files that have encountered some form of password protection or encryption security measures. These errors are not resolved unless the proper passwords or encryption keys are placed in the Password Bank. For more information, see Password Bank.

Note: When investigating publish errors, if you see five password protected errors associated with an .MSG file, but the email and all of its contents opens and displays correctly in the viewer, it means a password-protected container was attached to the email.

Password Protected FileThese errors are exclusive to non-container files that have encountered some form of password protection or encryption security measures. These errors are not resolved unless the proper passwords or encryption keys are placed in the Password Bank. For more information, see Password Bank.
Relativity Field ConfigurationThese errors represent issues with Field Mapping during publish jobs. They are usually associated to a specific setting like length or an Associative Object Type. When encountered, the field settings should be resolved according to the error message and resolved.
UnsupportedRelativity Processing has determined that these files are unsupported and was unable to obtain metadata or text from them. These files can be published to your workspace, but they may be inaccessible from the Viewer.

Details modal

You can open the Details modal of a file by clicking Expand details icon to see uncompressed file and content metadata not visible by default in the Files view.

The Details modal provides you with supplemental information about errors that have occurred on records during discovery and publish.

Discovered files details view

You can also see a summary and history of all Processing Errors and retries in this modal. When you click the Processing Errors tab, you're presented with the following breakdown of the current errors and error history of the selected file:

Discovered files details view

  • The Error History section represents all errors that have ever occurred on a file. This acts as a timeline of the record’s errors, showing when they occurred, what they were about, and if any are still active. This includes errors resulting from retries of previous errors and contains category, phase, date/time, and message information. All times are kept in UTC format.
  • The Error Summary section displays a count of all active errors along with their associated category and phase. This is especially important when investigating errors relating to container files, as there can be many associated to the parent container during file extraction. This helps determine the level of impact the issue has as it may affect many files originating from it.

Pivotable error fields

By default, all relevant processing error fields are available to group by and pivot on in the Current Errored Files and All Errored Files views of the Files tab.

(Click to expand)

For descriptions of all the fields available for Pivot, see the Files tab.

Document-level error workflow

ClosedSee details on document-level processing errors

The default Current Document Errors view in the Document Errors tab provides the following fields:

The Document Errors view provides the following fields:

  • File ID - the number value associated with the error file in the database.
  • Current Status - the current status of the error. You'll see any of the following values here:
    • In progress - the error is in progress for being ignored or retried.
    • Ready to retry - you're able to retry the error. You can only take action on an error if it has a status of Ready to Retry.
    • Unresolvable - you can't resolve the error by attempting to retry it.
    • Ignored - you ignored the error.
    • Retried - you retried the error and it is finished retrying.
    • In queue - the error is in the processing queue for retry.
    • Canceled - the error has been canceled by virtue of the processing job being canceled.
    • Ignored; Canceled - Relativity ignored the error and canceled the processing set.
    • Resolved - Relativity successfully retried the error after you addressed it, whether you did so by repairing the file outside of Relativity, providing a password to the password bank, or fixing an environmental issue.
  • Current Category - the system-assigned category of the error. This will display one of the following values:
    • Application Issue - the error was most likely caused by a third-party exception or issue outside of Relativity's control, or by a Relativity code problem.
    • Can't Open File or Part of File - the error occurred because an issue inside the file itself prevented Relativity from opening it.
    • Environmental Issue - the error was caused by an issue with the worker, whether it has to do with hardware, software, network connectivity, server configuration, etc.
    • Failed Import - the error was due to an issue with the Relativity import action.
    • General - the error, although analyzed already, still doesn't have enough information to be assigned a category.
    • Password Protected - the error occurred because the file is encrypted with a password that is not a valid entry in the Password Bank.
    • Potential Corrupt File - the error is most likely due to the file being corrupt.
    • Relativity Configuration - the error is due to an issue with the admin user settings in Relativity.
    • Uncategorized - the error does not have a corresponding category.
  • Current Error Message - the cause and nature of the error. When you click this message, you're taken to the error details layout, where you can use the error console to address the error. For more information, see Processing error workflow.
  • Source Location - the source path in which the file that has the error resides.
  • Custodian - the custodian associated with the data source containing the file on which the error occurred.
  • Processing Set - the name of the processing set in which the error occurred.
  • Processing Data Source - the data source containing the file on which the error occurred.
  • Number of Retries - the number of times a user has clicked Retry on this error.
  • Republish required - the error must be retried in order to be successfully published.

Note the following details about document-level errors:

  • If a processed file contains more than one error, only the error that was first encountered on that file and currently has a status of ready to retry is listed in the document errors view.

  • When you retry the error listed, you are actually retrying all of the other errors that occurred on that file, as well, even though they aren't currently visible in the view.

  • If you successfully retry the error listed, then the next ready-to-retry error on that file will be displayed.

  • The Number of Retries field reflects the number of times a user has clicked Retry on an individual error and not the grand total of retried errors associated with that file.

File error actions

Action can be taken on file errors from the Processing Set page or from the mass operations available on the Files tab.

Processing Set error retry

You can retry file errors within the Processing Set by using the Retry File Errors button located under the Processing Set console on the right-hand side of the page.

A confirmation message pops up reminding you of the errors you are about to retry. Click Retry to proceed or Cancel to return to the processing set layout.

Only file errors with a high chance of success will be retried. The probability of success is determined by the error category associated with the file. Error categories such as Corruption or Password Protection will not be retried as they are not likely to be resolved without manual intervention (for example, adding passwords or replacing a corrupt file). A full list of what will and will not be retried can be found below:

Error

Category

Included in

Retry Button

Corrupt ContainerNo
Corrupt FileNo
Could Not IdentifyNo
EnvironmentalYes
File Read/Write IssueYes
Missing AttachmentYes
Missing child items due to password protectionNo
Missing Extracted TextYes
Missing File MetadataYes
Partially Corrupted ContainerNo
Password Protected ContainerNo
Password Protected FileNo
Relativity Field ConfigurationNo
UnsupportedNo

Files tab error actions

From the Files tab, you can take action on your errored files through the mass operations view.

Mass operationDescription
Export as CSVThis exports the list of processing errors as a CSV file.
Republish

Gives you the option of republishing the files that the errors occurred on. Once you resolve the errors listed, you can use this option, and if the republish is successful, the files will be available in the Documents list, and no errors will be displayed in the Current Errored Files view.

For details on how to republish files from the Files tab, see Republishing files.

Retry Errors

This action provides the ability to resolve issues occurring during discover and publish. These issues can be found on the Current Errored Files and All Errored Files views within the Files tab. For details on how to retry errors from the Deleted Documents view in the Files tab, see Retrying delete errors.

  • You must have Edit permissions on the Discovered Files object to be able to retry file errors.

  • Note the following regarding retrying errors:

    • Auto-publish is not enabled when you retry errors. If any discover or text extraction errors are resolved, you will need to manually publish them into your workspace by navigating back to the processing set and clicking the Retry button.

    • Not all errors reported in the discovery process can be resolved. This is expected as processing reports all issues it encounters through an error.

    • You should always resolve all publish errors as these errors represent data not in review.

    • If an error occurs on a loose file during discovery, Relativity still attempts to publish it. For example, if a Password Protected error occurs on a PDF during discovery, that file still has the ability to be published in its current state. The resulting record may have metadata and/or extracted text missing depending on the issue, but it can still be referenced during review.

    • Relativity automatically retries all publish errors for a set when any error within that set is retried.

    • Multiple retry attempts cannot be worked on simultaneously. If a secondary retry is submitted while the initial one is still in progress, the second retry will wait in a queue until the first retry is completed.

    • Only errors with an Error Status of Not Resolved can be submitted in a retry job.

Ignore Errors

This provides the ability to set a file’s Error Status to Ignored, which will remove it from the Current Errored Files view. The record will still be visible in the All Files and All Errored Files views.
Undo Ignore ErrorsThis provides the ability to set a file’s Error Status field back to its original value after it had previously been ignored.
Download / Replace
  • This provides the ability to download a file to your local machine for investigation. It will also provide the ability to replace an original file with a new version that has been fixed during error remediation.

  • Note the following regarding downloading and replacing files:

    • you can only download / replace a single file at a time.

    • you can only perform these actions on files with an Error Status of Not Resolved.

    • there is no file size limitation when downloading files.

    • there is a one-gigabyte file size limitation when uploading replacement files.

    • performing a replacement of a file will automatically retry its associated errors once completed.

    • after uploading a new document, when you select Replace & Retry, the native file is updated before you republish.

    • the retry action for job errors will only retry errors in a Ready to Retry state.

For more information on the Download / Replace mass action, see Download and Replace on the Files tab page.

The following mass operations are available:

Common workflows

Identifying and resolving errors

You have completed discovery or publish on your processing set and noticed that it had encountered some errors. You want to investigate and resolve those errors quickly so you can get all possible data into your workspace. Starting from your processing set, perform the following steps:

  1. On the right-hand side of the page under Links, select File Errors to go directly to the Current Errored Files view on the Files tab. Automatic filtering takes you to the files in the previously viewed processing set.

  2. On the Files tab you can optionally filter down on the errored files that are the most important to resolve. Some common filters are the following:

    • Error Category - to group issues of a similar type.

    • Error Phase - to group issues that occurred during a particular part of Processing.

    • Custodian -if you have a priority Custodian that requires all records to be investigated first.

    • Sort Date - to retry files within the matter's relevant date range.

  3. Once a group of records is identified to resolve, select the Retry Errors mass action to begin the process. Alternatively, you can retry all errored files without filtering.

  4. You can now track your progress of the error retry through the processing set page’s progress bar or by navigating to the Worker Monitoring page in Home mode.

For more information on Worker Monitoring, see Processing Administration.

Replacing a corrupted file

Sometimes, files reach processing in a corrupted state. Here is a workflow to replace corrupted files with non-corrupted versions so you can get the most out of your data. This works on encrypted documents as well.

Note: For more information on replacement considerations, see Download / Replace.

Starting from your processing set, perform the following steps:

  1. On the right-hand side of the page under Links, select File Errors to go directly to the Current Errored Files view on the Files tab. Automatic filtering takes you to the errored files in the previously viewed processing set.

  2. Locate the file you need to replace. Common techniques are:

    • Filter Error Category for Corrupted File or Corrupted Container.

    • Filter by specific file names.

    • Filter for specific error messages.

  3. Select the appropriate checkbox on the left-hand column of the view.

  4. Select the Download / Replace option in the mass action menu.

  5. From here, two options are available:

    • to inspect and/or repair your container, select the download button

    • once you are in possession of your replacement container, drag and drop it into the modal or select browse for files to locate your container

  6. Once the replacement file has been added to the modal, it automatically uploads to Relativity. A quick verification process will let you know if any issues were found or if there were any significant differences between the original and replaced files.

  7. Select the Replace & Retry button to complete the replacement and retry any Discovery related errors.

Note: When replacing a file, the metadata associated with the new file overwrites any metadata associated with the original file. For example, if a file had an Author of Steve Bruhle in the original file, but has an Author of Dave Crews in the replaced file, the metadata in Relativity will have Dave Crews filled out.

Error processing job: SyntaxError: Unexpected token u in JSON at position #482

Hi there,

I am using Bull 2.6.5 (Redis 3.2.4) and Node 6.9.4 across about 30 containers each running a single queue consumer process with concurrency = 2.

Problem I'm Running Into
Over time my queues start to fill up with invalid jobs that Bull does not remove, and I start to see many errors in the logs:

When these errors occur the process is not killed. Instead these jobs just repeatedly get re-attempted and never removed. I think because the error is happening at the point where the Job object would actually be created, so it is not able to handle this like a standard job failure.

When I just checked I had about ~6000 jobs in this state.

What I've Done to Debug
In node:

  • When I print out the contents of inside of I see that in these failure cases it is always either or .

In redis-cli:

How to reproduce
I do not have a solid code-based reproduction process. I'm not sure why I end up with jobs with the empty data as I mentioned above. I'm not doing any manual redis queue manipulation in my actual code, but I believe you would be able to reproduce the problem by setting one of your jobs in your queue to be a hash map with just a single entry. Something like:

Workaround
My current workaround here, which I just got setup, is to proactively check for this invalid job state in , craft a with the same Id (and no data), and then return a rejected promise to remove this job from the queue (since it is invalid).

This appears to properly clean up these bugged jobs when they're encountered, but I understand it may not be an appropriate solution for the library. I'd be happy to help contribute something more appropriate, but I'd be curious to get @manast opinions on this whole situation.

Is there additional useful information could I provide to help narrow this down?

Processing errors

You could encounter a variety of errors while executing your processing job. To see a list of all processing errors for the workspace, navigate to the Processing Errors tab. From this tab, you can sort on the Message column to compare the errors returned with the errors displayed on the Document and/or Job Exception Report. You can also edit errors by adding notes about the error.

You can identify an error in the All Processing Errors view through any of the following fields:

Message The cause and nature of the error; for example, Unable to recurse embedded objects. Office document is password protected.
Processing phase

The state of the processing job when the error occurred:

  • Discovering files
  • Extracting data
  • Publishing files
Error status

The current state of the error:

  • In progress
  • Ready to retry
  • Unresolvable
  • Retried
  • Ignored
Error type

The type of error:

  • Document error occurs inside an individual file after the processing set that retrieved that file sent it through as part of the processing job. This type of error does not stop the job from completing. The job continues until the processing error limit is reached.
  • Job error stops the processing set from being submitted to the processing engine. This means that the processing job won't start until the error is resolved and the job is attempted again.
Processing set The processing set containing the file in error.
NotesAny notes added to describe the error
NameThe number identifier of the error as assigned by the processing server
Identified file type The file type of the document as it was identified by the processing engine. This is only populated for document-level errors. If the file type is unknown, this will display a value of Unknown.
Document file location The file path of the document in error. Click this link to open the file.
Container ID The ID of the container from which the file in error was directly extracted, as identified by the processing engine. If you have a zip file with a .pst file inside it, and the .pst file that contains an .msg file, then the .msg file will have the container ID of the .pst, not the zip. This is because the .pst is the direct container of the .msg file.
Error created on The date/time stamp of when the error was created in the native imaging engine.

Common document errors

The following are common document errors:

  • The file you are attempting to process is password protected and no password was supplied.
  • Relativity either could not open the container file or could not read the objects embedded in the container file.
  • A file is encrypted and a password was incorrect or not supplied.
  • You attempt to map two processing fields to the same Relativity field.
  • You attempt to map a processing field to a Relativity field that has a different field type.

Common job errors

The following are common job errors:

  • The attempt to import the file failed.
  • The attempt to retrieve workspace fields from Relativity failed.

Reading individual processing errors

Any error that occurs during the processing job is visible in the Processing Errors tab. Click the message or name of any error in the All Processing Errors view to bring up its details.

The Error Details layout reflects the same fields found in the All Processing Errors view except for the following:

NameThe number identification of the error as automatically generated by the processing engine. This name is used on the Document object to associate an error with a document in Relativity.

Note: The links for the Document file location and Document folder location fields on the error won't work if the website(s) containing them isn't added as a trusted site.

Processing error configuration table value

During a processing job, the number of errors pulled from the processing server at one time is determined by the following configurable value in the configuration table:

Relativity.CoreProcessingErrorRetrievalInitialBatchSize1000The initial number of errors to return from the processing server in one call. If this fails, the number is cut in half until success or failure in returning one error occurs.

Ignoring errors

You have the option of ignoring errors you aren't going to resolve immediately and want to filter out of your list view. For example, ignoring errors is a viable option when an Office document is password protected and you don’t have the password.

You must have edit permissions for processing errors in order to ignore errors.

To ignore errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to ignore. You can only ignore errors with a status of Ready to retry.
  3. Move to the mass operation menu in the bottom left corner, select Ignore and click Go. You can only ignore errors through this mass operation.

  4. A confirmation appears stating, “X errors are eligible to be ignored. Once you ignore these errors you can’t retry them until you have un-ignored them.” Click Ok.
  5. Once the selected errors have been ignored, check the Error status column for each error in the view. This status should be Ignored.

Un-ignoring errors

You can un-ignore errors that you previously ignored so that you can have the option of retrying them. You must have edit permissions for processing errors in order to un-ignore errors.

To un-ignore errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to un-ignore. You can only un-ignore errors that have a status of Ignored.
  3. Move to the mass operation menu in the bottom left corner and select Un-ignore and click Go. You can only un-ignore errors through this mass operation.

  4. A confirmation appears stating, “X errors are eligible to be un-ignored. Once you un-ignore these errors, they will be ready to retry.” Click Ok.
  5. Once the selected errors have been un-ignored, check the Error status column for each error in the view. This status should be Ready to retry.

Retrying errors

Once you've resolved the errors that occurred in the documents that you previously attempted to process, you can return to the Errors tab in Relativity and retry those errors through the Retry mass operation. You must have edit permissions for processing errors to retry them.

Note: Not all errors are resolvable inside Relativity. For example, you'll have to handle password-protected files outside of Relativity in order to process them.

To mass retry one or more errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to submit for retry.
  3. Select Retry from the mass operations menu in the bottom left corner. You can only retry errors with a status of Ready for retry.
  4. Click Go.
    1. A retry is audited.
    2. If a retried error has a status of In progress, you can’t delete the processing set of that error.

    Note: Once an error retry is submitted to the processing engine for a single processing set, another retry job for the same set cannot be submitted until the first retry is complete. The subsequent retry is placed in the queue in Relativity to await retry.

Reading retry status for file discovery and publishing errors

When you retry errors generated during file publishing, only one error per workspace can be worked on at a time.

You'll see any of the following values in the Status field on the processing set as those errors are being retried.

Waiting The user executed a mass retry of discovery or publishing errors and the agent has not yet picked up the job.
Retry – Initializing The agent picked up the job but hasn’t yet submitted it.
<error message> The agent could not submit the job; the error message is shown here.
Retry – Discovering Files/Publishing FilesThe agent has submitted the job.
Retry – Published/Discovered; Retrieving errors The processing server is done discovering or publishing files, but Relativity is not done retrieving errors (whether or not there are any).
Retry – Published/Updating tables The processing server is done discovering or publishing files, and Relativity is updating tables.
Retry – Published/Discovered with errors The processing server is done discovering or publishing files and Relativity is done retrieving errors, and there was at least one error.
Retry complete The processing server is done discovering or publishing files and Relativity is done retrieving errors, and there were no errors.

Unresolvable errors

Errors that can’t be retried can’t be resolved and are marked by a status of Unresolvable.

Note the following:

  • Only job level errors can reach a status of Unresolvable; document level errors can’t be unresolvable.
  • Not all job level errors are unresolvable.

You can’t resolve an unresolvable error, but you can learn more about the cause of the error by opening it and referring to the Details field in the Processing Job Level Errors layout to see when it occurred and why.

Processing job failing repeatedly with "Failed. Reason: InternalServerError: We encountered an internal error. Please try again." #2496

Describe the bug
I am trying to run a Processing job and the job fails part way with this error message:

Failed. Reason: InternalServerError: We encountered an internal error. Please try again.

When I check the logs for the job in cloudwatch, the only logs I see are the print statements I added to help debug the issue, the last statement in the logs is only part way in a loop and there are no other messages / errors other than the when I tail the logs from where I launch the Processing job.

This has been happening over and over and I am not sure how to figure out what is wrong with the job.

To reproduce
Run processing job.

Expected behavior
Job completes without errors or shows where it failed with a helpful stack trace of what went wrong instead of repeatedly saying

Screenshots or logs

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.47.2
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
  • Framework version: N/A
  • Python version: 3.8
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): Y

Additional context
Running using a instance (metrics show plenty of room left in CPU, disk and memory usage) and a custom container. Job takes a shelve file, does some data stuff and then generates a bunch of small files that are written to disk (data read and written leverages paths backed by ProcessingInput and ProcessingOutput)

What I see in the console near the top of the job page:
sm-processing-fail

Error processing job - consider

Job server stops processing due to error, "Processing is disabled (queues globally switched off)" (4247672)

产品:
Identity Manager
8.1.5, 8.1.4, 8.1.3, 8.0.5, 8.1.2, 8.1.1, 8.1, 8.0.4, 8.0.3
Identity Manager On Demand
Hosted
主题:
Troubleshooting
文章历史记录:
创建时间: 8/22/2022
上次更新时间: 8/24/2022

Your Request will be reviewed by our technical reviewer team and, if approved, will be added as a Topic in our Knowledgebase.

您可以在附属支持站点上查找适用于戴尔*产品*的在线支持帮助。单击“继续”,转至适用于*产品*的正确支持内容和帮助。

© 2022 One Identity LLC. ALL RIGHTS RESERVED. Feedback 使用条款 隐私

One Identity门户不再支持IE 8、9和10,建议将您的浏览器升级到最新版本的Internet Explorer或Chrome。

要升级到IE 11,请单击此处

要升级到Chrome,请单击此处

如果继续使用IE 8、9和10,您将无法充分利用我们所有出色的自助服务功能。

Processing errors

You could encounter a variety of errors while executing your processing job. To see a list of all processing errors for the workspace, navigate to the Processing Errors tab. From this tab, you can sort on the Message column to compare the errors returned with the errors displayed on the Document and/or Job Exception Report. You can also edit errors by adding notes about the error.

You can identify an error in the All Processing Errors view through any of the following fields:

Message The cause and nature of the error; for example, Unable to recurse embedded objects. Office document is password protected.
Processing phase

The state of the processing job when the error occurred:

  • Discovering files
  • Extracting data
  • Publishing files
Error status

The current state of the error:

  • In progress
  • Ready to retry
  • Unresolvable
  • Retried
  • Ignored
Error type

The type of error:

  • Document error occurs inside an individual file after the processing set that retrieved that file sent it through as part of the processing job. This type of error does not stop the job from completing. The job continues until the processing error limit is reached.
  • Job error stops the processing set from being submitted to the processing engine. This means that the processing job won't start until the error is resolved and the job is attempted again.
Processing set The processing set containing the file in error.
NotesAny notes added to describe the error
NameThe number identifier of the error as assigned by the processing server
Identified file type The file type of the document as it was identified by the processing engine. This is only populated for document-level errors. If the file type is unknown, this will display a value of Unknown.
Document file location The file path of the document in error. Click this link to open the file.
Container ID The ID of the container from which the file in error was directly extracted, as identified by the processing engine. If you have a zip file with a .pst file inside it, and the .pst file that contains an .msg file, then the .msg file will have the container ID of the .pst, not the zip. This is because the .pst is the direct container of the .msg file.
Error created on The date/time stamp of when the error was created in the native imaging engine.

Common document errors

The following are common document errors:

  • The file you are attempting to process is password protected and no password was supplied.
  • Relativity either could not open the container file or could not read the objects embedded in the container file.
  • A file is encrypted and a password was incorrect or not supplied.
  • You attempt to map two processing fields to the same Relativity field.
  • You attempt to map a processing field to a Relativity field that has a different field type.

Common job errors

The following are common job errors:

  • The attempt to import the file failed.
  • The attempt to retrieve workspace fields from Relativity failed.

Reading individual processing errors

Any error that occurs during the processing job is visible in the Processing Errors tab. Click the message or name of any error in the All Processing Errors view to bring up its details.

The Error Details layout reflects the same fields found in the All Processing Errors view except for the following:

NameThe number identification of the error as automatically generated by the processing engine. This name is used on the Document object to associate an error with a document in Relativity.

Note: The links for the Document file location and Document folder location fields on the error won't work if the website(s) containing them isn't added as a trusted site.

Processing error configuration table value

During a processing job, the number of errors pulled from the processing server at one time is determined by the following configurable value in the configuration table:

Relativity.CoreProcessingErrorRetrievalInitialBatchSize1000The initial number of errors to return from the processing server in one call. If this fails, the number is cut in half until success or failure in returning one error occurs.

Ignoring errors

You have the option of ignoring errors you aren't going to resolve immediately and want to filter out of your list view. For example, ignoring errors is a viable option when an Office document is password protected and you don’t have the password.

You must have edit permissions for processing errors in order to ignore errors.

To ignore errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to ignore. You can only ignore errors with a status of Ready to retry.
  3. Move to the mass operation menu in the bottom left corner, select Ignore and click Go. You can only ignore errors through this mass operation.

  4. A confirmation appears stating, “X errors are eligible to be ignored. Once you ignore these errors you can’t retry them until you have un-ignored them.” Click Ok.
  5. Once the selected errors have been ignored, check the Error status column for each error in the view. This status should be Ignored.

Un-ignoring errors

You can un-ignore errors that you previously ignored so that you can have the option of retrying them. You must have edit permissions for processing errors in order to un-ignore errors.

To un-ignore errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to un-ignore. You can only un-ignore errors that have a status of Ignored.
  3. Move to the mass operation menu in the bottom left corner and select Un-ignore and click Go. You can only un-ignore errors through this mass operation.

  4. A confirmation appears stating, “X errors are eligible to be un-ignored. Once you un-ignore these errors, they will be ready to retry.” Click Ok.
  5. Once the selected errors have been un-ignored, check the Error status column for each error in the view. This status should be Ready to retry.

Retrying errors

Once you've resolved the errors that occurred in the documents that you previously attempted to process, you can return to the Errors tab in Relativity and retry those errors through the Retry mass operation. You must have edit permissions for processing errors to retry them.

Note: Not all errors are resolvable inside Relativity. For example, you'll have to handle password-protected files outside of Relativity in order to process them.

To mass retry one or more errors:

  1. Navigate to the Processing Errors tab.
  2. Select the check box next to each error you want to submit for retry.
  3. Select Retry from the mass operations menu in the bottom left corner. You can only retry errors with a status of Ready for retry.
  4. Click Go.
    1. A retry is audited.
    2. If a retried error has a status of In progress, you can’t delete the processing set of that error.

    Note: Once an error retry is submitted to the processing engine for a single processing set, another retry job for the same set cannot be submitted until the first retry is complete. The subsequent retry is placed in the queue in Relativity to await retry.

Reading retry status for file discovery and publishing errors

When you retry errors generated during file publishing, only one error per workspace can be worked on at a time.

You'll see any of the following values in the Status field on the processing set as those errors are being retried.

Waiting The user executed a mass retry of discovery or publishing errors and the agent has not yet picked up the job.
Retry – Initializing The agent picked up the job but hasn’t yet submitted it.
<error message> The agent could not submit the job; the error message is shown here.
Retry – Discovering Files/Publishing FilesThe agent has submitted the job.
Retry – Published/Discovered; Retrieving errors The processing server is done discovering or publishing files, but Relativity is not done retrieving errors (whether or not there are any).
Retry – Published/Updating tables The processing server is done discovering or publishing files, and Relativity is updating tables.
Retry – Published/Discovered with errors The processing server is done discovering or publishing files and Relativity is done retrieving errors, and there was at least one error.
Retry complete The processing server is done discovering or publishing files and Relativity is done retrieving errors, and there were no errors.

Unresolvable errors

Errors that can’t be retried can’t be resolved and are marked by a status of Unresolvable.

Note the following:

  • Only job level errors can reach a status of Unresolvable; document level errors can’t be unresolvable.
  • Not all job level errors are unresolvable.

You can’t resolve an unresolvable error, but you can learn more about the cause of the error by opening it and referring to the Details field in the Processing Job Level Errors layout to see when it occurred and why.

Error Processing Representative job

Primary City/State:

Phoenix, Arizona

Department Name:

Error Processing-Ref Lab

Work Shift:

Day

Job Category:

Revenue Cycle

Flexible start time anywhere from 6am-8am depending on schedule.

POSITION SUMMARY

The Error Processing Representative is primarily responsible for reviewing accessions that fall into the Error Processing workflow categories and for obtaining any, and all missing billing information that is required to finalize claims in a timely manner. The Error Processing Representative is responsible for managing a high volume of work on a daily basis. To accomplish our goals, the following essential functions will be used and all employees are responsible for supporting the company’s goals, mission, and values; following all company policies and procedures; providing quality customer service; and following safety standards in performing all aspects of the functions below.

CORE FUNCTIONS

  1. Researches information received from the client, from websites, and/or from previous billed claims to obtain information that is missing or invalid.

  2. Completes daily assignments and projects as instructed by the Supervisor, Assistant Supervisor, or QC/Trainer.

  3. Reconciles returned requests and accurately enters data into the system in a timely manner.

  4. Responds to phone calls from clients to resolve billing issues as well as makes outgoing calls as needed.

  5. Has a working knowledge of internal client’s duties and responsibilities.

  6. Assists other billing sub-departments as needed.

MINIMUM QUALIFICATIONS

  • High School Diploma/GED.

  • Three (3) years’ experience in Medical Billing, Medical Office, or related field.

  • Ability to work independently and as a team member.

  • Ability to work with high volumes of data.

  • Ability to meet deadlines.

PREFERRED QUALIFICATIONS

  • Previous experience in Medical Billing and Coding.

  • Medical coding and computer knowledge.

  • Additional related education and/or experience.

Banner Health is one of the largest, nonprofit health care systems in the country and the leading nonprofit provider of hospital services in all the communities we serve. Throughout our network of hospitals, primary care health centers, research centers, labs, physician practices and more, our skilled and compassionate professionals use the latest technology to make health care easier, so life can be better. The many locations, career opportunities, and benefits offered at Banner Health help to make the Banner Journey unique and fulfilling for every employee.

EOE/Female/Minority/Disability/Veterans Banner Health supports a drug-free work environment. https://www.bannerhealth.com/careers/eeo

Application currently closed
This job was posted on Sat Apr 16 2022 and expired on Fri Apr 29 2022.

Processing error workflow

This topic provides information on working with errors that may occur during processing jobs.

This page contains the following information:

Required security permissions

The following security permissions are required to perform actions on File Errors:

Object SecurityTab Visibility
  • Discovered File - View, Edit

  • Download and Replace files with processing errors

The following security permissions are required to perform actions on Job Errors:

Object SecurityTab Visibility
  • Job Error - View, Edit

  • Processing Error - View, Edit

For more information on permissions, see Workspace permissions.

Processing errors tabs

The Files and Job Errors tabs allow you to easily locate issues that may have occurred in any processing sets. These errors are notified to you through the processing set page upon job completion.

Files tab

The Files tab contains all error information associated to specific files that have occurred during the discovery, publish, and deletion phases of processing. The Current Errored Files and All Errored Files views are tailored to error workflow by only containing information pertaining to the errors that have occurred.

  • The Current Errored Files view displays all outstanding errors from processing jobs. This is your primary location for workflows like error retry, ignore, and file replacement.

  • The All Errored Files view is primarily utilized for historical reporting of errors from processing sets. This view displays any file that has encountered an error, regardless of whether it was resolved or not. It is a good reference to export an error report out of Relativity for a given collection or set of custodians.

For more information on these views, see Files tab.

Job Errors tab

The Job Errors tab contains all errors that occurred on processing sets run in your workspace. These errors are usually not associated with any specific files within a processing set, but rather to the entire set itself.

Job Error views

The Current Job Errors view in the Job errors tab displays all unresolved job errors while the All Job Errors view displays any job error that has occurred throughout the lifecycle of the matter. Both views contain the following fields:

  • Error Identifier - the unique identifier of the error as it occurs in the database. When you click this message, you are taken to the error details layout, where you can view the stack trace and other information. Note that for Unresolvable errors, the console is disabled because you can't take any actions on that error from inside Relativity. For more information, see Processing error workflow.
  • Error Status - the status of the error. This is most likely Unresolvable.
  • Message - the cause and nature of the error. For example, "Error occurred while trying to overlay deduplication details. Please resolve publish error or republish documents from data source below. DataSource Artifact Id: 1695700".
  • Custodian - the custodian associated with the data source containing the file on which the error occurred.
  • Processing Set - the name of the processing set in which the error occurred.
  • Data Source - the data source containing the file on which the error occurred.
  • Error Created On - the date and time at which the error occurred during the processing job.
  • Republish Required - the error must be retried in order to be successfully published.
  • Notes - any manually added notes associated with the error.

Note: Errors occurring during inventory are always represented as Job Errors. For more information, see Inventory Errors.

Job Error layout

Clicking on the job error identifier value brings you to the Job Error Details layout.

Note that the Error Actions console is disabled for unresolvable job errors, since you can't retry or ignore job errors the way you can document errors.

Job error details

To see the job error's stack trace, click on the Advanced tab of the error details layout and view the value in the Stack Trace field.

Job error stack trace

Job-level error workflow

Useful error field information

The following sections provide information on error-specific fields and views that you can use in your processing workflow.

Combined error fields

The Files tab displays a single error associated to a file.

This error displays through the Error Message, Error Category, Error Phase, and Error Status fields. However, multiple errors can be associated to a single file at the same time as issues can occur during different phases of Processing. Relativity determines the displayed error based on a set precedence of Processing phases that could potentially block content from being published. The precedence is as follows:

  • Delete – a document was deleted from Relativity, but encountered an issue, potentially affecting recalculation of deduplication.

  • Publish – a document was supposed to be promoted to review but encountered an error and was held back.

  • Discover –a file may have encountered an issue during expansion and may not have extracted a child record and/or associated metadata.

  • Text Extraction – a file encountered an issue during text extraction and is missing some or all associated text.

Error status information

The Error Status field provides information on where the file is in error remediation.

This is helpful to determine if any further actions are required on a file or to see if an error had ever occurred on a record. When a file has all its errors resolved, the Error Message, Error Category, and Error Phase fields no longer display any content, but the Error Status field keeps a status of Resolved to indicate that it was a record that initially encountered issues but have since been fixed. The statuses of errors are as follows:

  • Not Resolved – The error is still outstanding.

  • Resolving – The error was submitted or in the process of being retried.

  • Resolved – The error was resolved.

  • Ignored – The error was ignored. See File error actions.

Error Category list

The Error Category field provides insight into the nature of the errors that have occurred during processing.

The following table provides a listing of all values on the Error Category field, along with a description of what kinds of issues those values bring back if filtered.

Category nameDescription
Corrupt Container

These errors are exclusive to container files that have encountered corruption when attempting to open and locate files within the container itself. When containers have these errors associated to them, you will not see any extracted loose files. These errors are usually either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.

Corrupt FileThese errors are exclusive to non-container files that have found elements of corruption during Processing. These errors are either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.
Could Not IdentifyRelativity Processing was unable to identify the file during Discovery. This may indicate corruption in the file but was unable to be determined at the time of discovery.
EnvironmentalThese errors are caused by issues in the Relativity Processing environment. These should be retried and resolved when encountered.
File Read / Write FailureThese errors are a subset of Environmental issues specifically caused by file system issues. These should be retried and resolved when encountered.
Missing AttachmentAn attachment from a document or email was not able to be extracted from its file.
Missing File MetadataA file is missing a piece of metadata.
Missing Extracted TextThese errors represent issues that occurred during Text Extraction jobs that have caused a file to be missing some or all associated text. A specific root cause was unable to be readily identified, but they should be retried and resolved where possible.
Partially Corrupted ContainerThese errors are exclusive to container files that have encountered corruption during extraction of specific records. When containers have these errors associated to them, you may see some files extracted, but not all. These errors are usually either ignored or downloaded offline for an investigation on whether the corruption can be remediated and then subsequently replaced and retried.
Password Protected Container

These errors are exclusive to container files that have encountered some form of password protection or encryption security measures. These errors are not resolved unless the proper passwords or encryption keys are placed in the Password Bank. For more information, see Password Bank.

Note: When investigating publish errors, if you see five password protected errors associated with an .MSG file, but the email and all of its contents opens and displays correctly in the viewer, it means a password-protected container was attached to the email.

Password Protected FileThese errors are exclusive to non-container files that have encountered some form of password protection or encryption security measures. These errors are not resolved unless the proper passwords or encryption keys are placed in the Password Bank. For more information, see Password Bank.
Relativity Field ConfigurationThese errors represent issues with Field Mapping during publish jobs. They are usually associated to a specific setting like length or an Associative Object Type. When encountered, the field settings should be resolved according to the error message and resolved.
UnsupportedRelativity Processing has determined that these files are unsupported and was unable to obtain metadata or text from them. These files can be published to your workspace, but they may be inaccessible from the Viewer.

Details modal

You can open the Details modal of a file by clicking Expand details icon to see uncompressed file and content metadata not visible by default in the Files view.

The Details modal provides you with supplemental information about errors that have occurred on records during discovery and publish.

Discovered files details view

You can also see a summary and history of all Processing Errors and retries in this modal. When you click the Processing Errors tab, you're presented with the following breakdown of the current errors and error history of the selected file:

Discovered files details view

  • The Error History section represents all errors that have ever occurred on a file. This acts as a timeline of the record’s errors, showing when they occurred, what they were about, and if any are still active. This includes errors resulting from retries of previous errors and contains category, phase, date/time, and message information. All times are kept in UTC format.
  • The Error Summary section displays a count of all active errors along with their associated category and phase. This is especially important when investigating errors relating to container files, as there can be many associated to the parent container during file extraction. This helps determine the level of impact the issue has as it may affect many files originating from it.

Pivotable error fields

By default, all relevant processing error fields are available to group by and pivot on in the Current Errored Files and All Errored Files views of the Files tab.

(Click to expand)

For descriptions of all the fields available for Pivot, see the Files tab.

Document-level error workflow

ClosedSee details on document-level processing errors

The default Current Document Errors view in the Document Errors tab provides the following fields:

The Document Errors view provides the following fields:

  • File ID - the number value associated with the error file in the database.
  • Current Status - the current status of the error. You'll see any of the following values here:
    • In progress - the error is in progress for being ignored or retried.
    • Ready to retry - you're able to retry the error. You can only take action on an error if it has a status of Ready to Retry.
    • Unresolvable - you can't resolve the error by attempting to retry it.
    • Ignored - you ignored the error.
    • Retried - you retried the error and it is finished retrying.
    • In queue - the error is in the processing queue for retry.
    • Canceled - the error has been canceled by virtue of the processing job being canceled.
    • Ignored; Canceled - Relativity ignored the error and canceled the processing set.
    • Resolved - Relativity successfully retried the error after you addressed it, whether you did so by repairing the file outside of Relativity, providing a password to the password bank, or fixing an environmental issue.
  • Current Category - the system-assigned category of the error. This will display one of the following values:
    • Application Issue - the error was most likely caused by a third-party exception or issue outside of Relativity's control, or by a Relativity code problem.
    • Can't Open File or Part of File - the error occurred because an issue inside the file itself prevented Relativity from opening it.
    • Environmental Issue - the error was caused by an issue with the worker, whether it has to do with hardware, software, network connectivity, server configuration, etc.
    • Failed Import - the error was due to an issue with the Relativity import action.
    • General - the error, although analyzed already, still doesn't have enough information to be assigned a category.
    • Password Protected - the error occurred because the file is encrypted with a password that is not a valid entry in the Password Bank.
    • Potential Corrupt File - the error is most likely due to the file being corrupt.
    • Relativity Configuration - the error is due to an issue with the admin user settings in Relativity.
    • Uncategorized - the error does not have a corresponding category.
  • Current Error Message - the cause and nature of the error. When you click this message, you're taken to the error details layout, where you can use the error console to address the error. For more information, see Processing error workflow.
  • Source Location - the source path in which the file that has the error resides.
  • Custodian - the custodian associated with the data source containing the file on which the error occurred.
  • Processing Set - the name of the processing set in which the error occurred.
  • Processing Data Source - the data source containing the file on which the error occurred.
  • Number of Retries - the number of times a user has clicked Retry on this error.
  • Republish required - the error must be retried in order to be successfully published.

Note the following details about document-level errors:

  • If a processed file contains more than one error, only the error that was first encountered on that file and currently has a status of ready to retry is listed in the document errors view.

  • When you retry the error listed, you are actually retrying all of the other errors that occurred on that file, as well, even though they aren't currently visible in the view.

  • If you successfully retry the error listed, then the next ready-to-retry error on that file will be displayed.

  • The Number of Retries field reflects the number of times a user has clicked Retry on an individual error and not the grand total of retried errors associated with that file.

File error actions

Action can be taken on file errors from the Processing Set page or from the mass operations available on the Files tab.

Processing Set error retry

You can retry file errors within the Processing Set by using the Retry File Errors button located under the Processing Set console on the right-hand side of the page.

A confirmation message pops up reminding you of the errors you are about to retry. Click Retry to proceed or Cancel to return to the processing set layout.

Only file errors with a high chance of success will be retried. The probability of success is determined by the error category associated with the file. Error categories such as Corruption or Password Protection will not be retried as they are not likely to be resolved without manual intervention (for example, adding passwords or replacing a corrupt file). A full list of what will and will not be retried can be found below:

Error

Category

Included in

Retry Button

Corrupt ContainerNo
Corrupt FileNo
Could Not IdentifyNo
EnvironmentalYes
File Read/Write IssueYes
Missing AttachmentYes
Missing child items due to password protectionNo
Missing Extracted TextYes
Missing File MetadataYes
Partially Corrupted ContainerNo
Password Protected ContainerNo
Password Protected FileNo
Relativity Field ConfigurationNo
UnsupportedNo

Files tab error actions

From the Files tab, you can take action on your errored files through the mass operations view.

Mass operationDescription
Export as CSVThis exports the list of processing errors as a CSV file.
Republish

Gives you the option of republishing the files that the errors occurred on. Once you resolve the errors listed, you can use this option, and if the republish is successful, the files will be available in the Documents list, and no errors will be displayed in the Current Errored Files view.

For details on how to republish files from the Files tab, see Republishing files.

Retry Errors

This action provides the ability to resolve issues occurring during discover and publish. These issues can be found on the Current Errored Files and All Errored Files views within the Files tab. For details on how to retry errors from the Deleted Documents view in the Files tab, see Retrying delete errors.

  • You must have Edit permissions on the Discovered Files object to be able to retry file errors.

  • Note the following regarding retrying errors:

    • Auto-publish is not enabled when you retry errors. If any discover or text extraction errors are resolved, you will need to manually publish them into your workspace by navigating back to the processing set and clicking the Retry button.

    • Not all errors reported in the discovery process can be resolved. This is expected as processing reports all issues it encounters through an error.

    • You should always resolve all publish errors as these errors represent data not in review.

    • If an error occurs on a loose file during discovery, Relativity still attempts to publish it. For example, if a Password Protected error occurs on a PDF during discovery, that file still has the ability to be published in its current state. The resulting record may have metadata and/or extracted text missing depending on the issue, but it can still be referenced during review.

    • Relativity automatically retries all publish errors for a set when any error within that set is retried.

    • Multiple retry attempts cannot be worked on simultaneously. If a secondary retry is submitted while the initial one is still in progress, the second retry will wait in a queue until the first retry is completed.

    • Only errors with an Error Status of Not Resolved can be submitted in a retry job.

Ignore Errors

This provides the ability to set a file’s Error Status to Ignored, which will remove it from the Current Errored Files view. The record will still be visible in the All Files and All Errored Files views.
Undo Ignore ErrorsThis provides the ability to set a file’s Error Status field back to its original value after it had previously been ignored.
Download / Replace
  • This provides the ability to download a file to your local machine for investigation. It will also provide the ability to replace an original file with a new version that has been fixed during error remediation.

  • Note the following regarding downloading and replacing files:

    • you can only download / replace a single file at a time.

    • you can only perform these actions on files with an Error Status of Not Resolved.

    • there is no file size limitation when downloading files.

    • there is a one-gigabyte file size limitation when uploading replacement files.

    • performing a replacement of a file will automatically retry its associated errors once completed.

    • after uploading a new document, when you select Replace & Retry, the native file is updated before you republish.

    • the retry action for job errors will only retry errors in a Ready to Retry state.

For more information on the Download / Replace mass action, see Download and Replace on the Files tab page.

The following mass operations are available:

Common workflows

Identifying and resolving errors

You have completed discovery or publish on your processing set and noticed that it had encountered some errors. You want to investigate and resolve those errors quickly so you can get all possible data into your workspace. Starting from your processing set, perform the following steps:

  1. On the right-hand side of the page under Links, select File Errors to go directly to the Current Errored Files view on the Files tab. Automatic filtering takes you to the files in the previously viewed processing set.

  2. On the Files tab you can optionally filter down on the errored files that are the most important to resolve. Some common filters are the following:

    • Error Category - to group issues of a similar type.

    • Error Phase - to group issues that occurred during a particular part of Processing.

    • Custodian -if you have a priority Custodian that requires all records to be investigated first.

    • Sort Date - to retry files within the matter's relevant date range.

  3. Once a group of records is identified to resolve, select the Retry Errors mass action to begin the process. Alternatively, you can retry all errored files without filtering.

  4. You can now track your progress of the error retry through the processing set page’s progress bar or by navigating to the Worker Monitoring page in Home mode.

For more information on Worker Monitoring, see Processing Administration.

Replacing a corrupted file

Sometimes, files reach processing in a corrupted state. Here is a workflow to replace corrupted files with non-corrupted versions so you can get the most out of your data. This works on encrypted documents as well.

Note: For more information on replacement considerations, see Download / Replace.

Starting from your processing set, perform the following steps:

  1. On the right-hand side of the page under Links, select File Errors to go directly to the Current Errored Files view on the Files tab. Automatic filtering takes you to the errored files in the previously viewed processing set.

  2. Locate the file you need to replace. Common techniques are:

    • Filter Error Category for Corrupted File or Corrupted Container.

    • Filter by specific file names.

    • Filter for specific error messages.

  3. Select the appropriate checkbox on the left-hand column of the view.

  4. Select the Download / Replace option in the mass action menu.

  5. From here, two options are available:

    • to inspect and/or repair your container, select the download button

    • once you are in possession of your replacement container, drag and drop it into the modal or select browse for files to locate your container

  6. Once the replacement file has been added to the modal, it automatically uploads to Relativity. A quick verification process will let you know if any issues were found or if there were any significant differences between the original and replaced files.

  7. Select the Replace & Retry button to complete the replacement and retry any Discovery related errors.

Note: When replacing a file, the metadata associated with the new file overwrites any metadata associated with the original file. For example, if a file had an Author of Steve Bruhle in the original file, but has an Author of Dave Crews in the replaced file, the metadata in Relativity will have Dave Crews filled out.

Error processing job: SyntaxError: Unexpected token u in JSON at position #482

Hi there,

I am using Bull 2.6.5 (Redis 3.2.4) and Node 6.9.4 across about 30 containers each running a single queue consumer process with concurrency = 2.

Problem I'm Running Into
Over time my queues start to fill up with invalid jobs that Bull does not remove, and I start to see many errors in the logs:

When these errors occur the process is not killed. Instead these jobs just repeatedly get re-attempted and never removed. I think because the error is happening at the point where the Job object would actually be created, so it is not able to handle this like a standard job failure.

When I just checked I had about ~6000 jobs in this state.

What I've Done to Debug
In node:

  • When I print out the contents of inside of I see that in these failure cases it is always either or .

In redis-cli:

How to reproduce
I do not have a solid code-based reproduction process. I'm not sure why I end up with jobs with the empty data as I mentioned above. I'm not doing any manual redis queue manipulation in my actual code, but I believe you would be able to reproduce the problem by setting one of your jobs in your queue to be a hash map with just a single entry. Something like:

Workaround
My current workaround here, which I just got setup, is to proactively check for this invalid job state in , craft a with the same Id (and no data), and then return a rejected promise to remove this job from the queue (since it is invalid).

This appears to properly clean up these bugged jobs when they're encountered, but I understand it may not be an appropriate solution for the library. I'd be happy to help contribute something more appropriate, but I'd be curious to get @manast opinions on this whole situation.

Is there additional useful information could I provide to help narrow this down?

Processing job failing repeatedly with "Failed. Reason: InternalServerError: We encountered an internal error. Please try again." #2496

Describe the bug
I am trying to run a Processing job and the job fails part way with this error message:

Failed. Reason: InternalServerError: We encountered an internal error. Please try again.

When I check the logs for the job in cloudwatch, the only logs I see are the print statements I added to help debug the issue, the last statement in the logs is only part way in a loop and there are no other messages / errors other than the when I tail the logs from where I launch the Processing job.

This has been happening over and over and I am not sure how to figure out what is wrong with the job.

To reproduce
Run processing job.

Expected behavior
Job completes without errors or shows where it failed with a helpful stack trace of what went wrong instead of repeatedly saying

Screenshots or logs

System information
A description of your system. Please provide:

  • SageMaker Python SDK version: 2.47.2
  • Framework name (eg. PyTorch) or algorithm (eg. KMeans): N/A
  • Framework version: N/A
  • Python version: 3.8
  • CPU or GPU: CPU
  • Custom Docker image (Y/N): Y

Additional context
Running using a instance (metrics show plenty of room left in CPU, disk and memory usage) and a custom container. Job takes a shelve file, does some data stuff and then generates a bunch of small files that are written to disk (data read and written leverages paths backed by ProcessingInput and ProcessingOutput)

What I see in the console near the top of the job page:
sm-processing-fail

error processing job

Thematic video

Error Processing Run - Process Runner Help

1 Comments

Leave a Comment