ungdata_reshape_questionsfinner faktisk filedescription når den ikke er angitt som argument.LagKUBEogLagFilgruppegir informativ feilmelding dersom kuben/filgruppen ikke er satt opp i accessmake_godkjent_folder()bruker korrekt tilkoblingsfunksjon
- Sikrer at spv_tmp er oppdatert før SPVFLAGG settes. Dersom flaggkolonner er satt i snutter må spv_tmp oppdateres først.
- Holder nå 9 utenfor serieprikking. Dette vil (kan) gi noe mer serieprikking av tidsserier som er kortere enn hele kuben.
- For fix_geo_special settes tall nå eksplisitt til NA. Dette fordi de tidligere utløste serieprikking pga stattol dersom de ble stående som 0.
- Fikset en liten bug i filtrering av merget teller-nevner-tabell som gav utslag når enkeltår skulle filtreres bort.
- Liten justering av ungdata-spesialfunksjoner
- Bugfix i generate_tmp_prednevner når årganger skulle filtreres ut
- Rydding i dospecial_handling, unngår å hente ut data mer enn nødvendig fra duckdb.
- Implementert mulighet for å kopiere fil over til lokal tempdisk før innlesing. Foreløpig bare aktuelt for UNGDATA hvor gevinsten er stor pga antall kolonner i filen.
- Option
khfunctions.localread_patternsangir regex-mønstre som matcher de originalfilene som skal behandles på denne måten.
- Option
- Implementert funksjonalitet for å sette unike startår i kombinasjonsindikatorer fra ungdata.
- Bugfix
- do_write_stata_file brukte en utfaset skrivefunksjon.
- Fjernet utfaset argument
codebooklogimake_table_from_original_file classify_columnstar høyde for overlapp mellom verdi- og dimensjonskolonner. Dersom det finnes en .f-kolonne regnes kolonnen som verdi og ikke dimensjon.
- Lagt til funksjon for å aggregere ungdatafiler i tilfeller der flere indikatorer skal slås sammen til en
- Fikset det slik at fildumper automatisk legges i environment ved lagring av CSV eller parquet-format.
- Dersom format = "envir", lagres ingen output til disk, f.eks.
dumps = list(RSYNT1pre = "envir")
- Dersom format = "envir", lagres ingen output til disk, f.eks.
- i compute_new_value_from_rowsums ble val-definisjoner for implisitte nuller feil for filgruppe BEF_GKny, da denne er håndtert tidligere i løypen.
- Fjerning av flere barnesykdommer
- Fjerning av barnesykdommer i 2.0.0
- Nyheter på norsk fra nå
- Prikking: spv_tmp = 2 (stort sett ved nevner = 0) holdes ikke utenfor personvernprikking lenger (issue178)
- Dekningsgrad bydel/levekårssone sjekkes nå konsekvent mot kommune (og ikke mot bydel). (issue177)
- Lagt til nye kvalitetskontroller for kube (issue173)
- Implementert ny pakke
khtoolsmed generisk funksjonalitet brukt på tvers av produksjonsapparatet- Tilkobling og generelle databasefunksjoner (
duckdb_FUNKSJONSNAVN) duckdb_save_file()eksporterer csv/parquet direkte fra duckdb (issue168)- Konsollmeldinger renset ved hjelp av
msg()ogheader() - config-globals inneholder globale defaultinnstillinger brukt på tvers av systemet, men som leses inn og dermed kan endres pakkevis
- khtools har egne options som skal brukes på tvers, men som ikke endres. Disse brukes som "khtools.optionsnavn"
- Pakken ikke lenger avhengig av orgdata da funksjonaliteten er flyttet til khtools
- Tilkobling og generelle databasefunksjoner (
- Filnavn og batchdato droppet fra duckdbnavnet for å unngå akkumulering av db-filer lokalt
- Outputfiler skrives direkte fra duckdb (issue168)
- Kolonnetyper blir satt eksplisitt ved eksportering av ALLVISfil (issue176)
- Parametertabeller som
KnrHarm,GeoKoderogLKS_STARTAARleses direkte inn i duckdb (issue169) classify_columns()implementert for å klassifisere kolonner som dimensjoner/verdier/hjelpekolonner- wrappere: identify_values, identify_nonvalues, and identify_dimensions
do_special_handling- Argumentene dt og duck utfaset, da denne bare kjører med duckdb fra nå.
- Leser og skriver til duckdb, med argumentenbe
tablenameogdt_namesom definerer tabell i duckdb og tabellnavn i kodemiljøet- For R or STATA, table =
tablenamehentes fra duckdb og lagret som data.table objektdt_namei kodemiljøet
- For R or STATA, table =
- Data skrives alltid tilbake til duckdb som tabell =
tablename - Kan motta SQL-spørringer som input. Da må accessfeltet merkes med
<SQL>(jobber direkte på tabell =tablenamei duckdb med con =duckdb_con) - Alle Rsynt-punkter bruker nå data fra duckdb og skriver tilbake (om ikke snutten er skrevet med SQL).
find_filedesignskriver til duckdb i stedet for å bruke tmp-fil (issue174)- Mer informative feilmeldinger
- All RODBC-kode erstattet med DBI
- graveyard2 opprettet, for funksjoner som ikke lenger brukes etter overgang til duckdb
- graveyard3 opprettet, for funksjoner erstattet av khtools
- Implementert duckdb-løsning for LagFilgruppe (issue158)
- Prosessering og stabling av originalfiler gjøres i duckdb, som minimerer minnebehov
- Originaliler som ikke er parquet leses inn i R og encoding fikses før filen skrives til duckdb
- Alle kolonner gjøres om til tekst ved innlesing
- Omkoding med kodebok gjøres nå i duckdb, og kodebokloggen ligger der (issue172)
RSYNT1er flyttet ut av read_original_file, slik at parquetfiler kan leses direkte inn i duckdb- Rensing, omdøping og konvertering av kolonnetyper skjer nå i duckdb
RSYNT_PRE_FGLAGRINGbruker nå duckdb, slik at filer ikke må leses inn i minnet (dersom snutt er SQL)
- Prosessering og stabling av originalfiler gjøres i duckdb, som minimerer minnebehov
- Partisjonerte versjoner av BEF_GKny er utfaset, siden det går raskt å filtrere en sortert filgruppe direkte ved innlesing.
load_and_format_files()leser filgruppene direkte inn i duckdb- All innlesing er nå lagt i en felles gren (tidligere flere grener avhengig av type fil og om det var satt opp filfiltre)
- All formattering skjer i duckdb
- Sikrer at kolonner som skal leses som integer får riktig format
do_harmonize_geoer oversatt til sql og kjøres i duckdb, med add_fylke som valgfritt argument- Aggregering og standardisering oversatt til sql og gjøres i duckdb.
- "SPVFLAGG" blir nå satt i KUBE (tidligere i ALLVIS) for å være tilgjengelig for kvalitetskontroll
- BUFFER er utfaset siden filer ikke holdes i minnet lenger
- LKS startår gjorde ingenting ikke dersom bydelsstart eller DK2020 ikke var satt opp (issue175)
- Fikset skrivefeil i
save_filedump_if_requestedsom ikke fungerte for csv-output. - Rettet bug i find_redesign hvor merge_cbpart_to_full brukte en keyed tabell, som gjorde mergen ustabil og feil.
- Critical bugfix in add_missing_lks()
- Added function to handle bydel/kommune levels with single lks. LKS is added as a copy of the parent level. For invalid zones which is not a single zone, a censored copy of the parent level is added. (issue170)
- Increased min version of duckdb to 1.5.5 due to new functionality.
- Critical bugfix in censoring, which did not remove numbers if censoring was turned off. Numbers are still removed for other reasons, e.g. LKS_STARTAAR, and need to get the appropriate SPVFLAGG.
- Expand qualcontrol cube (issue164)
report_value_totalsadded to aggregation check, to see if underniv is much smaller than overnivreport_n_uniqe_geonivreports number of unique geocodes, and number of kommune/bydel with lks data
do_format_cube_columnsnow also adds censor-columns- Added new columns
geoprikketandmanuellprikket fix_geo_specialmoved to after do_format_cube_columns, to be able to setgeoprikket = 1fix_geo_specialno longer removes values (issue152). Values removed IF STATA-censoring is active.- Increased R-dependency to >= 4.1
- Made
read_stata_filesmore robust against edge cases where string columns were saved asstring_viewon some (virtual) machines, which could not be read by R.
control_rate_lksgains a N_lks column indicating number of lks in each strata (issue160)- remove option to force R censoring
- Skip secondary censoring when no triangles are specified
- Implement
LKS_STARTAARtable
- adjusted to new access format, removed connection to deprecated khlogg.
- activated qualcontrol functions lks
- Options is now checked every time the package is loaded (issue143)
- Fixed error message when original file not found (issue151, @raniets)
- Implemented new function
set_threads. InLagKUBE/LagFilgruppe, data.table and collapse now utilizes more threads to speed up processing. do_special_handlinggains dt_name argument, to ensure correct name in code environment.- More functions are now operating by reference, avoiding unneccessary copies. This reduces memory requirements drastically!
- FILFILTRE::nykolsMERGE deactivated, (replaced with special function for BEFVEKST)
- added function
merge_cols_by_reference(), which adds columns without making full copies of data
- Implemented new function
do_filter_KUILwhich filters out only needed levels of KJONN/UTDANN/INNVKAT/LANDBAK. This reduces size of files drastically before further processing (issue153) FinnFellesTabimproved for run time and memory requirement.- Implemented
do_handle_coverage, which censor strata on bydel/levekaar based on coverage. (issue150) - Information on secondary censoring is collected in one column (issue144)
- Added more columns with information on censoring (orgprikket, dekningprikket) (issue149)
do_filfiltre_kollapsdelernow checks for existing totals before aggregating, added safety against double counting- Added function
add_leadyear_befvekst, removing the need for nykolsmerge which was only used for this. - When loading filegroups, colorder is set according to standarddimensions.
merge_teller_nevnerworks more by reference, relying less on copies of data- Implemented
duckdbfor temporary storage of data (issue157)- Required filegroups and temporary products are now stored in a on-the-fly duckdb database
- Several new functions operated directly on tables in duckdb, meaning they do not need to be loaded into RAM
- Implemented function
control_rate_lks(issue128) compare_geolevelsnow handles all geolevels, and comparisons are made for all strata within each overlevel (issue128)control_meis_rate()filters out rows with diff = 0, and prints full table if nrow <= 10control_standardization()no longer warns that standardization is not towards last year for UNGDATA files.
- Remove get(col)/mget(col), replace with
dt[[col]],.SDcolsorenv =, whichever is most relevant - Implemented data.table::set() several places replacing := or let()
- New function
print_console_messagewraps around cat(), but ensure that output is printed in full
- Fixed message on recoding when no values recoded
- In read_org_excel, sheet name is now forced as character
make_godkjent_folder()now also runsqualcontrol::check_friskvik()and warns if errors are detected. Need qualcontrol > v1.3.8
- Removed unneccessary columns NORM.n/.a/.f which were generated when MALTALL == TELLER/RATE
- ALLVIS output is also saved as .parquet, with appropriate schema to ensure column types
- Some code changes to ensure functionality in chained operations (now written on separate lines) due to interactions between data.table v1.18 and collapse.
- reintroduced csv-output for QC-files by request.
find_periods()gains a check that unique years must be >= period, to avoid generating e.g. 3-year moving averages when only 2 years exist in the file.- setting of initial spv_tmp moved to after aggregation to periods, else it was removed which caused problems with censoring.
lagkube_cleanup()checks if guardfile exists before trying to delete it- Deprecated csv-output for QC-file
- Removed circular dependencies
- Critical bugfix in
format_excel_and_csv_files()misinterpreting TomRadSlutt.
- Started working on implementing duckdb to reduce memory requirement
- Special_handling no longer use api to fetch available scripts for rsynt-points, but read a compiled list on github. This avoids issues with github api-limit when running many files.
- Updated
generate_specific_friskvik_indicators()to read parquet files and use correct date tag. - Updated dependencies and package versions
- Implementing
do_censor_primary_secondary()to replace R and STATA censoring, handling both primary and secondary censoring. ** In ACCESS, limits are fetched from either PRIKK_T/N/STATTOL, or Stata_PRIKK_T/N/STATTOL. Triangles are always fetched from Stata_nabopr*-columns. ** For now, the new routine is activated when using R-censoring for testing. ** Secondary censoring is performed by new functiondo_naboprikk() - spv_tmp is now set throughout the data processing, and can be used to set SPVFLAGG. However, flag variables are often changed in postprocess code, currently hindering this implementation.
- Stacking of original files to filegroup is more effective by using list and only data.table::rbindlist() at the end
- Implementing ALLVISFILTER, providing the functionality to filter ALLVIS files before writing, to e.g. remove rows/columns (but keep them in the QC-file)
check_if_format_is_okdid not report correctly when there were problems.- Fixed SPVFLAGG when RATE is NA due to TELLER & NEVNER == 0. Correct SPVFLAGG is 2, but became 3 due to TELLER < limit. Now RATE/TELLER/NEVNER.f is set to 2 for these rows.
- Added more informative message when strata missing from standard files, causing problems with standardization.
do_balance_teller_nevnernow only handles TELLER/NEVNER, as crude RATE is calculated after.- Primary censoring is only done when flag variable == 0
- LagKUBE and LagFilgruppe now ensures nb-NO.UTF-8 locale, and resets the original locale when the function exits.
do_harmonize_geosets year if not in parameters, making it available for LagFilgruppe and rsynt.- Added secondary censoring triangles for levekaarsone
- For rsynt-scripts read from github, case-insensitive matching has been enabled in case of autocorrect in access. A warning will be given if any url is not exact matching.
- Requested dumps in R format are saved as
.parquet - Further information on censoring included in QC-file
LagQCKubenow uses collapse::join for merging.- FORMAT for original files are now identified with case-insensitive matching
- When saving partitioned population files, use case-insensitive matching with file name "BEF_GKny".
- Remove object
testfrom LagQCKube, which was only meant for testing.
do_special_handlinggains a...argument to pass e.g. filedescription for RSYNT1 into the code_env- KnrHarm is added to parameters for filegroup, to be available for special handling
- Explicitly set column order when making table from original file
- Added use.names to rbindlist for file groups, to ensure correct matching if rows are not in the same order
set_filter_agenow accepts when tellerfile do not have age columns- FILFILTRE is identified with case-insensitive matching to ensure match when spelling differs
- SetKubeParameters/SetFilgruppeParameters now disconnects from ACCESS after collecting parameters
- Fixed order of functions when selecting population data set.
- KODEBOK_LOGG also outputs TYPE and OMK (new value) columns
- crude RATE is set to NA if NaN (teller == 0 and nevner == 0) or Inf (nevner == 0).
- Critical fix in filtering of population file, where age filter was not correct. Now
alderfilterwill be NULL if amin/amax equals limits set in config file.
- Critical fix where wrong name of dataset was provided when generating new columns after merging teller/nevner
- Bugfix in selecting information for setting implicit null values when filegroup was BEF_GKa (due to different value names)
write_population_filegroupsaves BEF_GKny filegroups as a partitioned dataset on age, year, and geolevel. This facilitates faster read times when needing only part of the file.- A separate read function,
read_population_file()which reads the partitioned BEF_GKny is added toload_filegroup_to_buffer()** Checks whether filters exists for age, year or geoniv (lks yes/no). Selects the correct dataset and reads the needed parts only. - In
do_write_parquet(), switched from removing attributes to converting to arrow table before writing SetFilgruppeParameters()andSetKubeParameters()automatically generate the parameters object in the global environment
- Fixed code to delete org_geo_codes from Global Environment in
lagfilgruppe_cleanup() - Fixed installation of pq package in generated stata .do-file, which crashed when new version was available
compute_new_value_from_row_sum()replacesLeggTilSumFraRader(). Fixed bug when too many columns were added, causing problems when aggregating the file.control_meis_rate()now outputs all diffs on country level, as well as top 5 diffs in both directions.
- NORM correctly renamed when MALTALL is not TELLER or RATE
- Quality control of CUBE failed when all sumTELLER or sumNEVNER was missing
- Added qualcontrol and orgdata to remotes in description
- In
LagKUBE(), moved Rsynt SLUTTREDIGER to after POSTPROSESS add_crude_rate()calculates crude RATE and helper columns, replacing compute_new_value_from_formula.control_meis_rate()added to cube quality control, reports min/max of the MEIS/RATE ratio.
- In
compute_new_value_from_formula(), the .n column is now set to the maximum of the included .n-columns. Previously it was set to 1. - In
estimate_prednevner(), missing years are filtered out to ensure match between observed and predicted teller. Otherwise, MEIS was not correct.
- Improved reporting on 99-codes in
LagFilgruppe
- Fixed problem in
do_clean_ALDERwhere checks for ALDERl > ALDERh was performed on character columns, not numeric.
- Implemented
.parquet-format. ** Filegroups are now saved as.parquetinstead of.rds. **read_filegroup()now prioritizes reading.parquetfiles **read_original_file()gains the ability to read.parquetfiles (FORMAT must be 'PARQUET'), in preparation for when orgdata produces.parquetfiles. ** QC-files and are saved as.parquet(in addition to.csv, which will be deprecated in the future) ** Full cube files are saved as.parquetinstead of.rds**do_stata_processing()now write and read.parquet, unless values contains norwegian characters (due to encoding). **generate_stata_do_file()is adapted to use the correct files. This saves much time previously spent writing and reading large.dta-files check_encoding()scans character columns in filegroups looking for garbled characters indicating that files are read with wrong encoding.- A written log of LagFilgruppe is saved, similar to what is saved when running LagKUBE. Both files are now only saved when write = TRUE.
generate_specific_friskvik_indicators()can be used to generate friskvik indicators without remaking the cube file. Useful if any friskvik spec is wrong or a new indicator must be set up.update_production_folder()can be used to update the production folder, removing outdated and adding missing files from DATERT- Qualcontrol of filegroup
**
analyze_cleanlog()moved into qualcontrol **warn_geo_99checks if any 99-codes is found, and tell the user to scroll up to find the original codes. - Qualcontrol of cube
**
control_censoringchecks if any value are outside the limits **control_standardizationchecks if standardization year is the most recent **control_aggregation,compare_geolevels, andcompare_bydel_kommunechecks if aggregation between geographical levels is ok for sumteller and sumnevner (if in file) ** Print a summary message after all checks are performed
fix_column_name_pre_stata()is no longer failing due to encodingcontrol_censoring()handles cases where no values were outside the limits resulting in nrow = 0
- Remove all use of
with = FALSEinside data.table - In
LagFilgruppe, codebooklog and cleanlog is only saved if write = TRUE
- Fix: cube files was saved with wrong name
- Fixed do_reshape_var when measure.var is not provided and should be all non-id-columns
- Added Censor_type to parameters
- Started implementing qualcontrol in LagKUBE, set to inactive until done
- Critical bugfix in set_recode_filter_filfiltre() when a cube use > 1 FILFILTRE
- Reactivated FF_RSYNT1 making it possible to add custom R code to a filefilter
- Critical bugfix when fixing column names pre and post stata handling
- Added parameter
removebufferto LagKUBE, to disable removing of original files during processing. This is useful for dev work, and will be necessary when implementing KUBEFAMILIE which is using the same files.
- Implemented function
fix_befgk_spelling()to handle problem with inconsistent spelling of BEF_GK[..], which sometimes appears as BEF_Gk. - Fixed bugs in LagFilgruppe where illegal values for INNVKAT was corrected, and illegal values for ALDER was mixed up with AAR.
analyze_cleanlog()now only gives a warning if illegal values are found.- Fixed bug when converting column names before STATA export
- Fixed bugs in setting SPVFLAGG
- Fixed bugs in friskvik and write_cube_output
- Fixed bug in year-filter when loading filegroups
- Ignore case in KUBER::REFVERDI, allowing both siste and SISTE
- Fixed predictionfilter to make sure last period is always used for moving averages, and GEOniv=='L' is added if missing in REFVERDI
- Fix auto-update function
- Structured code as an R package
- Adding documentation
- Rewritten code to consistently use data.table instead of data.frame, and implementing collapse functions where possible
- To reduce memory usage, several functions now alter the data by reference instead of overwriting or copying objects.
- Cleaned up code ** removed inactive parts ** refactored into functions with more descriptive names ** moved relevant comments into roxygen where possible.
- Collecting all parameters used in
LagFilgruppe()andLagKUBE()into one listparameters, reducing the number of arguments passed to internal functions.` - Adding
stop()where the previous code just printed a message that something was wrong and continued. - Implementing saving of filedumps (pre/post) in
do_special_handling, as well as inaggregate_to_periods,do_censor_cube,do_reshape_var, andrecode_columns_with_codebook
- Consistently read in files as data.table
- Collect information on recoding with kodebok and cleaning of columns into tables in R, instead of writing continuously to ACCESS
- Rows deleted via codebook are deleted while stacking original files, limiting the data siz
- Adding analysis of the column cleaning, which forces a crash if something is not ok
- Output will always be both to NYESTE and to DATERT folder, removing the need for argument
versjonert - KODEBOK_LOGG is now written as a date tagged CSV instead of writing to ACCESS::KODEBOKLOGG
- FILGRUPPE_SJEKK is written as a date tagged CSV, and replaces ACCESS::INNLESLOGG
- Output are stored as a list "RESULTAT" in the global environment
- Changed default for write to TRUE
- All filegroups are now consistently filtered to only use relevant TAB1/2/3, AAR, and ALDER based on ACCESS input. This reduces data size, and avoids unnecessary memory use. ** AAR prior to AAR_START is removed ** ALDER < min and > max ALDER required is removed ** TABs are filtered to only keep what is provided in TABX(_0)
- Implemented standardization to multiyear-periods. ** In ACCESS, the last year of the period must be provided. It could be given as AARl=='XXXX', but it is more intuitive to use AARh =='XXXX'
- Filegroups are always read from NYESTE, simplifying the code.
- When initially loading filegroups, the files will always be read from disk and never from BUFFER.
- Moved R and STATA censoring into the same function
do_censor_cube** R censoring is no longer removing values, only setting flags