Skip to content

Latest commit

 

History

History
456 lines (349 loc) · 27.1 KB

NEWS.md

File metadata and controls

456 lines (349 loc) · 27.1 KB

khfunctions 2.0.9 (2026-11-09)

  • ungdata_reshape_questions finner faktisk filedescription når den ikke er angitt som argument.
  • LagKUBE og LagFilgruppe gir informativ feilmelding dersom kuben/filgruppen ikke er satt opp i access
  • make_godkjent_folder() bruker korrekt tilkoblingsfunksjon

khfunctions 2.0.8 (2026-10-30)

  • Sikrer at spv_tmp er oppdatert før SPVFLAGG settes. Dersom flaggkolonner er satt i snutter må spv_tmp oppdateres først.

khfunctions 2.0.7 (2026-10-29)

  • Holder nå 9 utenfor serieprikking. Dette vil (kan) gi noe mer serieprikking av tidsserier som er kortere enn hele kuben.
  • For fix_geo_special settes tall nå eksplisitt til NA. Dette fordi de tidligere utløste serieprikking pga stattol dersom de ble stående som 0.

khfunctions 2.0.6 (2026-10-26)

  • Fikset en liten bug i filtrering av merget teller-nevner-tabell som gav utslag når enkeltår skulle filtreres bort.
  • Liten justering av ungdata-spesialfunksjoner
  • Bugfix i generate_tmp_prednevner når årganger skulle filtreres ut
  • Rydding i dospecial_handling, unngår å hente ut data mer enn nødvendig fra duckdb.

khfunctions 2.0.5 (2026-10-23)

  • Implementert mulighet for å kopiere fil over til lokal tempdisk før innlesing. Foreløpig bare aktuelt for UNGDATA hvor gevinsten er stor pga antall kolonner i filen.
    • Option khfunctions.localread_patterns angir regex-mønstre som matcher de originalfilene som skal behandles på denne måten.
  • Implementert funksjonalitet for å sette unike startår i kombinasjonsindikatorer fra ungdata.
  • Bugfix
    • do_write_stata_file brukte en utfaset skrivefunksjon.
    • Fjernet utfaset argument codebooklog i make_table_from_original_file
    • classify_columns tar høyde for overlapp mellom verdi- og dimensjonskolonner. Dersom det finnes en .f-kolonne regnes kolonnen som verdi og ikke dimensjon.

khfunctions 2.0.4 (2026-10-20)

  • Lagt til funksjon for å aggregere ungdatafiler i tilfeller der flere indikatorer skal slås sammen til en
  • Fikset det slik at fildumper automatisk legges i environment ved lagring av CSV eller parquet-format.
    • Dersom format = "envir", lagres ingen output til disk, f.eks. dumps = list(RSYNT1pre = "envir")

khfunctions 2.0.3 (2026-10-15)

  • i compute_new_value_from_rowsums ble val-definisjoner for implisitte nuller feil for filgruppe BEF_GKny, da denne er håndtert tidligere i løypen.

khfunctions 2.0.2 (2026-10-14)

  • Fjerning av flere barnesykdommer

khfunctions 2.0.1 (2026-10-14)

  • Fjerning av barnesykdommer i 2.0.0

khfunctions 2.0.0 (2026-10-14)

  • Nyheter på norsk fra nå

Ny eller endret funksjonalitet som gir endret output i LagKUBE()

  • Prikking: spv_tmp = 2 (stort sett ved nevner = 0) holdes ikke utenfor personvernprikking lenger (issue178)
  • Dekningsgrad bydel/levekårssone sjekkes nå konsekvent mot kommune (og ikke mot bydel). (issue177)
  • Lagt til nye kvalitetskontroller for kube (issue173)

Generelle endringer

  • Implementert ny pakke khtools med generisk funksjonalitet brukt på tvers av produksjonsapparatet
    • Tilkobling og generelle databasefunksjoner (duckdb_FUNKSJONSNAVN)
    • duckdb_save_file() eksporterer csv/parquet direkte fra duckdb (issue168)
    • Konsollmeldinger renset ved hjelp av msg() og header()
    • config-globals inneholder globale defaultinnstillinger brukt på tvers av systemet, men som leses inn og dermed kan endres pakkevis
    • khtools har egne options som skal brukes på tvers, men som ikke endres. Disse brukes som "khtools.optionsnavn"
    • Pakken ikke lenger avhengig av orgdata da funksjonaliteten er flyttet til khtools
  • Filnavn og batchdato droppet fra duckdbnavnet for å unngå akkumulering av db-filer lokalt
  • Outputfiler skrives direkte fra duckdb (issue168)
    • Kolonnetyper blir satt eksplisitt ved eksportering av ALLVISfil (issue176)
  • Parametertabeller som KnrHarm, GeoKoder og LKS_STARTAAR leses direkte inn i duckdb (issue169)
  • classify_columns() implementert for å klassifisere kolonner som dimensjoner/verdier/hjelpekolonner
    • wrappere: identify_values, identify_nonvalues, and identify_dimensions
  • do_special_handling
    • Argumentene dt og duck utfaset, da denne bare kjører med duckdb fra nå.
    • Leser og skriver til duckdb, med argumentenbe tablename og dt_name som definerer tabell i duckdb og tabellnavn i kodemiljøet
      • For R or STATA, table = tablename hentes fra duckdb og lagret som data.table objekt dt_name i kodemiljøet
    • Data skrives alltid tilbake til duckdb som tabell = tablename
    • Kan motta SQL-spørringer som input. Da må accessfeltet merkes med <SQL> (jobber direkte på tabell = tablename i duckdb med con = duckdb_con)
    • Alle Rsynt-punkter bruker nå data fra duckdb og skriver tilbake (om ikke snutten er skrevet med SQL).
  • find_filedesign skriver til duckdb i stedet for å bruke tmp-fil (issue174)
    • Mer informative feilmeldinger
  • All RODBC-kode erstattet med DBI
  • graveyard2 opprettet, for funksjoner som ikke lenger brukes etter overgang til duckdb
  • graveyard3 opprettet, for funksjoner erstattet av khtools

LagFilgruppe

  • Implementert duckdb-løsning for LagFilgruppe (issue158)
    • Prosessering og stabling av originalfiler gjøres i duckdb, som minimerer minnebehov
      • Originaliler som ikke er parquet leses inn i R og encoding fikses før filen skrives til duckdb
      • Alle kolonner gjøres om til tekst ved innlesing
    • Omkoding med kodebok gjøres nå i duckdb, og kodebokloggen ligger der (issue172)
    • RSYNT1 er flyttet ut av read_original_file, slik at parquetfiler kan leses direkte inn i duckdb
    • Rensing, omdøping og konvertering av kolonnetyper skjer nå i duckdb
    • RSYNT_PRE_FGLAGRING bruker nå duckdb, slik at filer ikke må leses inn i minnet (dersom snutt er SQL)
  • Partisjonerte versjoner av BEF_GKny er utfaset, siden det går raskt å filtrere en sortert filgruppe direkte ved innlesing.

LagKUBE

  • load_and_format_files() leser filgruppene direkte inn i duckdb
    • All innlesing er nå lagt i en felles gren (tidligere flere grener avhengig av type fil og om det var satt opp filfiltre)
    • All formattering skjer i duckdb
    • Sikrer at kolonner som skal leses som integer får riktig format
  • do_harmonize_geo er oversatt til sql og kjøres i duckdb, med add_fylke som valgfritt argument
  • Aggregering og standardisering oversatt til sql og gjøres i duckdb.
  • "SPVFLAGG" blir nå satt i KUBE (tidligere i ALLVIS) for å være tilgjengelig for kvalitetskontroll
  • BUFFER er utfaset siden filer ikke holdes i minnet lenger

Bugfix

  • LKS startår gjorde ingenting ikke dersom bydelsstart eller DK2020 ikke var satt opp (issue175)
  • Fikset skrivefeil i save_filedump_if_requested som ikke fungerte for csv-output.
  • Rettet bug i find_redesign hvor merge_cbpart_to_full brukte en keyed tabell, som gjorde mergen ustabil og feil.

khfunctions 1.2.7 (2026-08-10)

  • Critical bugfix in add_missing_lks()

khfunctions 1.2.6 (2026-08-04)

  • Added function to handle bydel/kommune levels with single lks. LKS is added as a copy of the parent level. For invalid zones which is not a single zone, a censored copy of the parent level is added. (issue170)
  • Increased min version of duckdb to 1.5.5 due to new functionality.

khfunctions 1.2.5 (2026-06-16)

  • Critical bugfix in censoring, which did not remove numbers if censoring was turned off. Numbers are still removed for other reasons, e.g. LKS_STARTAAR, and need to get the appropriate SPVFLAGG.

khfunctions 1.2.4 (2026-06-12)

  • Expand qualcontrol cube (issue164)
    • report_value_totals added to aggregation check, to see if underniv is much smaller than overniv
    • report_n_uniqe_geoniv reports number of unique geocodes, and number of kommune/bydel with lks data

khfunctions 1.2.3 (2026-06-05)

  • do_format_cube_columns now also adds censor-columns
  • Added new columns geoprikket and manuellprikket
  • fix_geo_special moved to after do_format_cube_columns, to be able to set geoprikket = 1
  • fix_geo_special no longer removes values (issue152). Values removed IF STATA-censoring is active.
  • Increased R-dependency to >= 4.1
  • Made read_stata_files more robust against edge cases where string columns were saved as string_view on some (virtual) machines, which could not be read by R.

khfunctions 1.2.2 (2026-06-03)

  • control_rate_lks gains a N_lks column indicating number of lks in each strata (issue160)
  • remove option to force R censoring
  • Skip secondary censoring when no triangles are specified
  • Implement LKS_STARTAAR table

khfunctions 1.2.1 (2026-05-29)

  • adjusted to new access format, removed connection to deprecated khlogg.
  • activated qualcontrol functions lks

khfunctions 1.2.0 (2026-05-28)

General changes

  • Options is now checked every time the package is loaded (issue143)
  • Fixed error message when original file not found (issue151, @raniets)
  • Implemented new function set_threads. In LagKUBE/LagFilgruppe, data.table and collapse now utilizes more threads to speed up processing.
  • do_special_handling gains dt_name argument, to ensure correct name in code environment.
  • More functions are now operating by reference, avoiding unneccessary copies. This reduces memory requirements drastically!
  • FILFILTRE::nykolsMERGE deactivated, (replaced with special function for BEFVEKST)
  • added function merge_cols_by_reference(), which adds columns without making full copies of data

LagKUBE improvements

  • Implemented new function do_filter_KUIL which filters out only needed levels of KJONN/UTDANN/INNVKAT/LANDBAK. This reduces size of files drastically before further processing (issue153)
  • FinnFellesTab improved for run time and memory requirement.
  • Implemented do_handle_coverage, which censor strata on bydel/levekaar based on coverage. (issue150)
  • Information on secondary censoring is collected in one column (issue144)
  • Added more columns with information on censoring (orgprikket, dekningprikket) (issue149)
  • do_filfiltre_kollapsdeler now checks for existing totals before aggregating, added safety against double counting
  • Added function add_leadyear_befvekst, removing the need for nykolsmerge which was only used for this.
  • When loading filegroups, colorder is set according to standarddimensions.
  • merge_teller_nevner works more by reference, relying less on copies of data
  • Implemented duckdb for temporary storage of data (issue157)
    • Required filegroups and temporary products are now stored in a on-the-fly duckdb database
    • Several new functions operated directly on tables in duckdb, meaning they do not need to be loaded into RAM

Quality control for CUBE

  • Implemented function control_rate_lks (issue128)
  • compare_geolevels now handles all geolevels, and comparisons are made for all strata within each overlevel (issue128)
  • control_meis_rate() filters out rows with diff = 0, and prints full table if nrow <= 10
  • control_standardization() no longer warns that standardization is not towards last year for UNGDATA files.

Code cleaning

  • Remove get(col)/mget(col), replace with dt[[col]], .SDcols or env =, whichever is most relevant
  • Implemented data.table::set() several places replacing := or let()
  • New function print_console_message wraps around cat(), but ensure that output is printed in full

khfunctions 1.1.6 (2026-02-25)

Bugfix

  • Fixed message on recoding when no values recoded
  • In read_org_excel, sheet name is now forced as character

khfunctions 1.1.5 (2026-01-20)

New features

  • make_godkjent_folder() now also runs qualcontrol::check_friskvik() and warns if errors are detected. Need qualcontrol > v1.3.8

Other changes

  • Removed unneccessary columns NORM.n/.a/.f which were generated when MALTALL == TELLER/RATE

khfunctions 1.1.4 (2026-01-14)

New features

  • ALLVIS output is also saved as .parquet, with appropriate schema to ensure column types

Other changes

  • Some code changes to ensure functionality in chained operations (now written on separate lines) due to interactions between data.table v1.18 and collapse.
  • reintroduced csv-output for QC-files by request.
  • find_periods() gains a check that unique years must be >= period, to avoid generating e.g. 3-year moving averages when only 2 years exist in the file.
  • setting of initial spv_tmp moved to after aggregation to periods, else it was removed which caused problems with censoring.

khfunctions 1.1.3 (2026-01-05)

Bugfix and other changes

  • lagkube_cleanup() checks if guardfile exists before trying to delete it
  • Deprecated csv-output for QC-file
  • Removed circular dependencies

khfunctions 1.1.2 (2025-12-16)

Bugfix

  • Critical bugfix in format_excel_and_csv_files() misinterpreting TomRadSlutt.

khfunctions 1.1.1 (2025-12-10)

New features

  • Started working on implementing duckdb to reduce memory requirement

Bugfix Other changes

  • Special_handling no longer use api to fetch available scripts for rsynt-points, but read a compiled list on github. This avoids issues with github api-limit when running many files.
  • Updated generate_specific_friskvik_indicators() to read parquet files and use correct date tag.
  • Updated dependencies and package versions

khfunctions 1.1.0 (2025-11-20)

New features

  • Implementing do_censor_primary_secondary() to replace R and STATA censoring, handling both primary and secondary censoring. ** In ACCESS, limits are fetched from either PRIKK_T/N/STATTOL, or Stata_PRIKK_T/N/STATTOL. Triangles are always fetched from Stata_nabopr*-columns. ** For now, the new routine is activated when using R-censoring for testing. ** Secondary censoring is performed by new function do_naboprikk()
  • spv_tmp is now set throughout the data processing, and can be used to set SPVFLAGG. However, flag variables are often changed in postprocess code, currently hindering this implementation.
  • Stacking of original files to filegroup is more effective by using list and only data.table::rbindlist() at the end
  • Implementing ALLVISFILTER, providing the functionality to filter ALLVIS files before writing, to e.g. remove rows/columns (but keep them in the QC-file)

khfunctions 1.0.22 (2025-10-28)

Bugfix and other changes

  • check_if_format_is_ok did not report correctly when there were problems.
  • Fixed SPVFLAGG when RATE is NA due to TELLER & NEVNER == 0. Correct SPVFLAGG is 2, but became 3 due to TELLER < limit. Now RATE/TELLER/NEVNER.f is set to 2 for these rows.
  • Added more informative message when strata missing from standard files, causing problems with standardization.
  • do_balance_teller_nevner now only handles TELLER/NEVNER, as crude RATE is calculated after.
  • Primary censoring is only done when flag variable == 0

khfunctions 1.0.20 (2025-10-07)

New features

  • LagKUBE and LagFilgruppe now ensures nb-NO.UTF-8 locale, and resets the original locale when the function exits.
  • do_harmonize_geo sets year if not in parameters, making it available for LagFilgruppe and rsynt.
  • Added secondary censoring triangles for levekaarsone
  • For rsynt-scripts read from github, case-insensitive matching has been enabled in case of autocorrect in access. A warning will be given if any url is not exact matching.
  • Requested dumps in R format are saved as .parquet
  • Further information on censoring included in QC-file

Bugfixes and other changes

  • LagQCKube now uses collapse::join for merging.
  • FORMAT for original files are now identified with case-insensitive matching
  • When saving partitioned population files, use case-insensitive matching with file name "BEF_GKny".
  • Remove object test from LagQCKube, which was only meant for testing.

khfunctions 1.0.19 (2025-09-18)

New features

  • do_special_handling gains a ... argument to pass e.g. filedescription for RSYNT1 into the code_env
  • KnrHarm is added to parameters for filegroup, to be available for special handling

Bugfixes and other changes

  • Explicitly set column order when making table from original file
  • Added use.names to rbindlist for file groups, to ensure correct matching if rows are not in the same order
  • set_filter_age now accepts when tellerfile do not have age columns
  • FILFILTRE is identified with case-insensitive matching to ensure match when spelling differs
  • SetKubeParameters/SetFilgruppeParameters now disconnects from ACCESS after collecting parameters

khfunctions 1.0.18 (2025-09-05)

Bugfix

  • Fixed order of functions when selecting population data set.

khfunctions 1.0.17 (2025-09-03)

Bugfix

  • KODEBOK_LOGG also outputs TYPE and OMK (new value) columns
  • crude RATE is set to NA if NaN (teller == 0 and nevner == 0) or Inf (nevner == 0).

khfunctions 1.0.16 (2025-09-02)

Bugfix

  • Critical fix in filtering of population file, where age filter was not correct. Now alderfilter will be NULL if amin/amax equals limits set in config file.

khfunctions 1.0.15 (2025-08-29)

Bugfix

  • Critical fix where wrong name of dataset was provided when generating new columns after merging teller/nevner
  • Bugfix in selecting information for setting implicit null values when filegroup was BEF_GKa (due to different value names)

khfunctions 1.0.14 (2025-08-26)

New features

  • write_population_filegroup saves BEF_GKny filegroups as a partitioned dataset on age, year, and geolevel. This facilitates faster read times when needing only part of the file.
  • A separate read function, read_population_file() which reads the partitioned BEF_GKny is added to load_filegroup_to_buffer() ** Checks whether filters exists for age, year or geoniv (lks yes/no). Selects the correct dataset and reads the needed parts only.
  • In do_write_parquet(), switched from removing attributes to converting to arrow table before writing
  • SetFilgruppeParameters() and SetKubeParameters() automatically generate the parameters object in the global environment

Bugfix

  • Fixed code to delete org_geo_codes from Global Environment in lagfilgruppe_cleanup()
  • Fixed installation of pq package in generated stata .do-file, which crashed when new version was available

khfunctions 1.0.13 (2025-08-20)

New features

  • compute_new_value_from_row_sum() replaces LeggTilSumFraRader(). Fixed bug when too many columns were added, causing problems when aggregating the file.
  • control_meis_rate() now outputs all diffs on country level, as well as top 5 diffs in both directions.

Bugfix

  • NORM correctly renamed when MALTALL is not TELLER or RATE
  • Quality control of CUBE failed when all sumTELLER or sumNEVNER was missing

Other changes

  • Added qualcontrol and orgdata to remotes in description

khfunctions 1.0.12 (2025-08-15)

New features

  • In LagKUBE(), moved Rsynt SLUTTREDIGER to after POSTPROSESS
  • add_crude_rate() calculates crude RATE and helper columns, replacing compute_new_value_from_formula.
  • control_meis_rate() added to cube quality control, reports min/max of the MEIS/RATE ratio.

Bugfix

  • In compute_new_value_from_formula(), the .n column is now set to the maximum of the included .n-columns. Previously it was set to 1.
  • In estimate_prednevner(), missing years are filtered out to ensure match between observed and predicted teller. Otherwise, MEIS was not correct.

Other changes

  • Improved reporting on 99-codes in LagFilgruppe

khfunctions 1.0.11 (2025-08-13)

Bugfix

  • Fixed problem in do_clean_ALDER where checks for ALDERl > ALDERh was performed on character columns, not numeric.

khfunctions 1.0.10 (2025-08-12)

New features

  • Implemented .parquet-format. ** Filegroups are now saved as .parquet instead of .rds. ** read_filegroup() now prioritizes reading .parquet files ** read_original_file() gains the ability to read .parquet files (FORMAT must be 'PARQUET'), in preparation for when orgdata produces .parquet files. ** QC-files and are saved as .parquet (in addition to .csv, which will be deprecated in the future) ** Full cube files are saved as .parquet instead of .rds ** do_stata_processing() now write and read .parquet, unless values contains norwegian characters (due to encoding). ** generate_stata_do_file() is adapted to use the correct files. This saves much time previously spent writing and reading large .dta-files
  • check_encoding() scans character columns in filegroups looking for garbled characters indicating that files are read with wrong encoding.
  • A written log of LagFilgruppe is saved, similar to what is saved when running LagKUBE. Both files are now only saved when write = TRUE.
  • generate_specific_friskvik_indicators() can be used to generate friskvik indicators without remaking the cube file. Useful if any friskvik spec is wrong or a new indicator must be set up.
  • update_production_folder() can be used to update the production folder, removing outdated and adding missing files from DATERT
  • Qualcontrol of filegroup ** analyze_cleanlog() moved into qualcontrol ** warn_geo_99 checks if any 99-codes is found, and tell the user to scroll up to find the original codes.
  • Qualcontrol of cube ** control_censoring checks if any value are outside the limits ** control_standardization checks if standardization year is the most recent ** control_aggregation, compare_geolevels, and compare_bydel_kommune checks if aggregation between geographical levels is ok for sumteller and sumnevner (if in file) ** Print a summary message after all checks are performed

Bugfixes

  • fix_column_name_pre_stata() is no longer failing due to encoding
  • control_censoring() handles cases where no values were outside the limits resulting in nrow = 0

Other updates

  • Remove all use of with = FALSE inside data.table
  • In LagFilgruppe, codebooklog and cleanlog is only saved if write = TRUE

khfunctions 1.0.9 (2025-07-02)

  • Fix: cube files was saved with wrong name

khfunctions 1.0.8 (2025-06-26)

  • Fixed do_reshape_var when measure.var is not provided and should be all non-id-columns
  • Added Censor_type to parameters
  • Started implementing qualcontrol in LagKUBE, set to inactive until done

khfunctions 1.0.7 (2025-06-24)

  • Critical bugfix in set_recode_filter_filfiltre() when a cube use > 1 FILFILTRE
  • Reactivated FF_RSYNT1 making it possible to add custom R code to a filefilter

khfunctions 1.0.6 (2025-06-23)

  • Critical bugfix when fixing column names pre and post stata handling
  • Added parameter removebuffer to LagKUBE, to disable removing of original files during processing. This is useful for dev work, and will be necessary when implementing KUBEFAMILIE which is using the same files.

khfunctions 1.0.5 (2025-06-20)

  • Implemented function fix_befgk_spelling() to handle problem with inconsistent spelling of BEF_GK[..], which sometimes appears as BEF_Gk.
  • Fixed bugs in LagFilgruppe where illegal values for INNVKAT was corrected, and illegal values for ALDER was mixed up with AAR.
  • analyze_cleanlog() now only gives a warning if illegal values are found.
  • Fixed bug when converting column names before STATA export

khfunctions 1.0.4 (2025-06-19)

  • Fixed bugs in setting SPVFLAGG
  • Fixed bugs in friskvik and write_cube_output
  • Fixed bug in year-filter when loading filegroups
  • Ignore case in KUBER::REFVERDI, allowing both siste and SISTE

khfunctions 1.0.1 (2025-06-17)

  • Fixed predictionfilter to make sure last period is always used for moving averages, and GEOniv=='L' is added if missing in REFVERDI
  • Fix auto-update function

khfunctions 1.0.0 (2025-06-16)

Summary of changes overall

  • Structured code as an R package
  • Adding documentation
  • Rewritten code to consistently use data.table instead of data.frame, and implementing collapse functions where possible
  • To reduce memory usage, several functions now alter the data by reference instead of overwriting or copying objects.
  • Cleaned up code ** removed inactive parts ** refactored into functions with more descriptive names ** moved relevant comments into roxygen where possible.
  • Collecting all parameters used in LagFilgruppe() and LagKUBE() into one list parameters, reducing the number of arguments passed to internal functions.`
  • Adding stop() where the previous code just printed a message that something was wrong and continued.
  • Implementing saving of filedumps (pre/post) in do_special_handling, as well as in aggregate_to_periods, do_censor_cube, do_reshape_var, and recode_columns_with_codebook

Changes in LagFilgruppe

  • Consistently read in files as data.table
  • Collect information on recoding with kodebok and cleaning of columns into tables in R, instead of writing continuously to ACCESS
  • Rows deleted via codebook are deleted while stacking original files, limiting the data siz
  • Adding analysis of the column cleaning, which forces a crash if something is not ok
  • Output will always be both to NYESTE and to DATERT folder, removing the need for argument versjonert
  • KODEBOK_LOGG is now written as a date tagged CSV instead of writing to ACCESS::KODEBOKLOGG
  • FILGRUPPE_SJEKK is written as a date tagged CSV, and replaces ACCESS::INNLESLOGG
  • Output are stored as a list "RESULTAT" in the global environment

Changes in LagKUBE

  • Changed default for write to TRUE
  • All filegroups are now consistently filtered to only use relevant TAB1/2/3, AAR, and ALDER based on ACCESS input. This reduces data size, and avoids unnecessary memory use. ** AAR prior to AAR_START is removed ** ALDER < min and > max ALDER required is removed ** TABs are filtered to only keep what is provided in TABX(_0)
  • Implemented standardization to multiyear-periods. ** In ACCESS, the last year of the period must be provided. It could be given as AARl=='XXXX', but it is more intuitive to use AARh =='XXXX'
  • Filegroups are always read from NYESTE, simplifying the code.
  • When initially loading filegroups, the files will always be read from disk and never from BUFFER.
  • Moved R and STATA censoring into the same function do_censor_cube ** R censoring is no longer removing values, only setting flags