Post Process Redevelopment Primer

(Updated as of 4/26/2022)

This primer will provide you the necessary documentation and tools in navigating through the changes in the CALPADS system as a result of the redesign of file submission and post processing. This page includes excerpts from CALPADS Flash 221 and other documentation and training sources.

Objectives and Required Changes

The objectives of the Post Process Redevelopment are to:

• Significantly reduce the time it takes to validate data within a file being submitted; and
• Significantly reduce the time it takes to post files to CALPADS.

In order to accomplish these objectives, the following system changes were necessary:

• Replace the single-lane file posting queue with a multi-lane file posting queue.
• Reduce the number of validations that occur upon file input by focusing on validating only data within the incoming file rather than comparing the data in the incoming file against data in the CALPADS Operational Data Store (ODS).
• Post a file only if it passes all input validations.
• Eliminate partial-post, and auto-post files only if it passes all input validations.
• Move validations that were previously conducted on input, but which require comparing data in the incoming file against data in the CALPADS ODS, to display after the file has posted in new data discrepancy reports, which will also include validations that will eventually become Certification Validation errors.
• Redesign the Statewide Student Identifier (SSID) assignment process to exclude the comparison of demographic data in the incoming file to demographic data in the ODS.
• Run anomaly detection throughout the day to provide immediate feedback regarding Multiple Identifiers (MID) anomalies.

Overview of Changes

File Submission

The File Submission process has been redeveloped in the CALPADS system to improve performance. This core function has been enhanced to free up user time, workload, and support error resolution in the data. The enhancement has removed partial post-processing and fails the file uploaded when input validations detect any records that do not meet quality specifications. Errors are displayed immediately to show the user which records need to be reviewed and corrected for a successful submission.

Legacy Workflow

Redesign Workflow

What does this mean for LEAs?

• Local file submission for Special Education Data System (SEDS) will continue to use API process
• File submissions to CALPADS will either Post all records or Fail entire file
  • LEAs will need to review Data Discrepancy reports for additional errors
• SIS Vendors could implement local file validations to reduce file failures
• Collaboration/Integration between SIS and LEAs will be CRITICAL
• SSID Process requires changes in local practices to ensure SSID integrity

File Submission Summary of Changes

• Multiple files can be uploaded regardless of file order/sort. However, we still recommend following prescribed sequence of submission to minimize avoidable file data discrepancies.
• Entire file rejected with one or more initial errors
• Retired SSIDs are replaced with "Kept" SSIDs during upload.
• Files with all passed records will automatically post
• Archive file functionality removed
• No more Partial Post (all or nothing)
• Additional errors are produced after data has been posted and must be checked and addressed if necessary, after post is complete

Changes to File Submission Statuses

Part of the change is the redesign of view submission file statuses. A file may be in one of the following possible states after the upload process or during the batch submission process:

Online Maintenance

CALPADS SSID Assignment

Currently, the acquisition of SSIDs is integrated in the submission of the Student Enrollment (SENR) file. Under the current process, LEAs upload an SENR that must include a local student identifier (ID) assigned by the LEA, and the SSID, if it is known. When SSIDs are unknown (for example, for new or transferring students), the SSID field is left blank. Based on the demographic data provided in the SENR, CALPADS conducts a search in the CALPADS ODS for students with blank SSID fields, and provides a list of potential candidates that match the demographics provided in the file. LEAs make SSID selections (existing or new) for students from the list, and then post the SENR records with the selected SSIDs for the selected students.

As described in the previous section, in order to increase performance of the file upload process, the search for potential matches against the CALPADS ODS has been eliminated as part of the new post process. Instead, CALPADS will now assign a new SSID to students with a blank SSID field (except when the local student ID matches a local student ID for the same LEA in the CALPADS ODS). Understanding that elimination of this functionality may result in an increase of MID anomalies, changes to local business processes that can mitigate the creation of MIDs are described below. In addition, the CDE will be implementing additional functionality following the initial release of the redeveloped post process, to enable LEAs to more easily search for SSIDs prior to submitting files to CALPADS.

New SSID Assignment Process

CALPADS will perform the same process regardless of file size for the new auto-assignment of SSIDs. That said, here are two workflow options LEAs may take depending on the volume of blank SSIDs. While Option 1 is preferable, it may not be feasible for LEAs that experience large numbers of new and transfer students on an ongoing basis. LEAs may also pursue both options, depending on the volume of transfers they experience in any given time period.

Option 1: Low Volume Transfers/New SSIDs – Student Demographic Search

1. Upon enrolling in an LEA, perform a student search in CALPADS Online Maintenance based on demographic criteria.
2. If an existing SSID is found, manually update the SSID in the Student Information System (SIS).
3. Generate the SENR extract from the SIS and upload to CALPADS.
4. If an existing SSID is NOT found, proceed with creating a new SSID by selecting “Create New SSID” from the Online Maintenance screen and CALPADS will assign a new SSID.
5. Download the SSID Extract to populate the SIS with the new SSIDs, as described below.

Option 2: High Volume Transfers/New SSIDs – SENR Batch File and MID Resolution

1. Generate the SENR extract from the SIS and upload to CALPADS, which includes blank SSID fields for new and transferring students.

2. For records where the SSID is blank, CALPADS then:

   a. Assigns an existing SSID where there ARE ODS records with a matching Local Student ID at the same LEA;

   b. Assigns a new SSID where there are no ODS records with a matching Local Student ID at the same LEA; then

   c. Automatically posts the file.

3. Once the SENR is posted, in order to avoid MIDs from being submitted to the Test Operations Management System (TOMS) in the nightly feed of CALPADS enrollments to the test vendor:

   a. Resolve MIDs on the same day that the SENR records are posted through MID Resolution, which is now updated every time new records are posted to the database. (NOTE: MIDs are now only matched on exact first and last name and date of birth, but will be enhanced in the near future to include probabilistic matching. The MID Resolution screen has been enhanced to display Parent/Guardian information to help identify the correct student.) b. If MIDs are not resolved on the same day, CALPADS Administrators should share MID reports with Testing Coordinators to ensure that students with possible MIDs are not tested until the MIDs are resolved.

4. Only once MIDs are resolved, download the SSID Extract to populate the SIS with new or replacement SSIDs that result from resolving MIDs, as described below.

New SSID Replacement Process

In the current system, when an SENR file is submitted with an existing SSID that has been retired, LEAs receive a warning that the SSID has been retired and that CALPADS will replace the retired SSID with the retained SSID. In the new system, CALPADS will continue to replace retired SSIDs with retained SSIDs, however, LEAs will not receive a notification. Therefore, LEAs will need to download the Replacement ID Extract to view the retired SSIDs that were replaced.

Loading New and Replacement SSIDs into the SIS

The SSID Extract is now downloaded by a date range that is based on the Enrollment Start Date and not based on job ID. Therefore, the SSID Extract may include SSIDs that already exist in the SIS, which may cause errors when trying to import the SSID Extract file into the SIS. To avoid this problem, LEAs may want to import the SSID Extract into the SIS using the Local Student ID as the key, which will then update blank SSIDs in the SIS. However, since each SIS works a little differently, it will be important to work with your SIS vendor for specific instructions on how to resolve any errors.

Importing the SSID Extract file into the SIS using the Local Student ID as the key, may also update SSIDs in the SIS that have been retired or replaced due to a MID merge, or cause an error. Therefore, it is also important to work with your SIS vendor to understand how to resolve any errors, or identify which SSIDs have been replaced.

Identifying the SSIDs for newly acquired students

There are 2 ways to determine the issued SSIDs for newly acquired students.

Through SSID Extract

When requesting the SSID extract, shorten your date range to a more recent period. Should the generated extract include all students, you can sort it by enrollment start date from newest to oldest. To do that, download the SSID Extract template (https://documentation.calpads.org/Support/FileTemplates/#extract-templates) and then follow instructions below.

Download the SSID Extract Template and follow the steps below in converting the extract.

1. Download SSID Extract Template.
2. Open the template.
3. A protected View message will appear at the top, you may click on Enable Editing.
4. Another security warning will appear at the top concerning a disable Macros. Click on Enable Content.
5. Click on Import SSID Extract File button.
6. Locate the downloaded SSID extract you requested from CALPADS.
7. Once you click on the file, the template will then gather the information from the extract and list the data extracted.
8. Sort the enrollment start date to show the most latest at the top to see the most recently created SSIDs.

Through Online Maintenance

1. Click on Online Maintenance.
2. Click on Student Data
3. Click on Search by Enrollment tab
4. Filter for desired schools
5. Under Select Status, Choose Currently Enrolled
6. Click Search
7. Scroll down to Results to see generated list.
8. Click on Local ID header to sort list until orange arrow switches to descending. You will see your new students listed at the top of the list with their SSIDs. (Assuming these are new students to your LEA and Local ID numbers are issued).

Summary of Suggestions for Changes to Local Business Processes

• Since the SSID Candidate List will no longer be generated upon uploading the SENR to CALPADS, LEAs should perform a manual search of students in CALPADS to determine whether or not a student already has an SSID, then manually update the SIS with any existing SSIDs prior to generating the SENR extract from the SIS.
• Since the new SSID assignment process may result in more MIDs, MIDs should be resolved on the same day the SENR is posted in order to avoid MIDs from being sent to TOMs in the nightly feed, and prior to downloading the SSID Extract to import into the SIS. To accommodate the need to resolve MIDs on the same day the SENR is posted, MID processing will occur on an ongoing basis throughout the day, and not just overnight.
• LEAs should be aware that the MID matching process will now only produce matches when there is an exact match on First Name, Last Name, and Birth Date; probabilistic matching will be implemented in a future release.
• Since users will no longer receive a warning when a retired SSID submitted in the SENR has been replaced with the replacement SSID, users should regularly download the SSID Replacement extract to determine which SSIDs have been retired within the SIS.
• Since the SSID Extract can only be generated using a date range based on Enrollment Start Date, and is no longer available by Job ID, there is a potential for the extract to include SSIDs that are already in the SIS, making it important to work with your SIS vendor who may have to modify import criteria to accommodate existing SSIDs in the import file.

Immediately Mitigating the Risk of MID Creation

The increase of MIDs that will initially occur could impact Initial and Summative ELPAC testing as well as Smarter Balanced Assessments; therefore, it is important to mitigate MID creation to the extent possible. Following the implementation of the redeveloped post process, the CDE recommends following the steps outlined in Option 1 and Option 2 described above. The key actions that LEAs can take to mitigate the creation of MIDs that are included in these options include:

1. Modifying local business processes to include a manual student search in CALPADS Online Maintenance to search for possible SSID matches for the student and manually enter those SSIDs into the SIS prior to generating and submitting the SENR to CALPADS. Do not send a SENR file with a blank SSID field for students transferring from public schools within California, if it can be avoided. While we recognize this process is manual and not ideal, especially for larger LEAs, it is an option immediately available to avoid MID creation.

2. Reviewing and resolving the MID anomalies on the same day that the SENR was posted if blank SSID fields are included in the SENR file. Doing so will minimize the number of MIDs being sent to TOMS in the nightly feed, and ensure the correct/permanent SSIDs are imported into the SIS.

3. Sharing MID Anomaly reports with assessment coordinators, if MIDs are not resolved on the same day that the enrollment file is posted, so they do not test the students until the MIDs have been resolved.