public/contra-costa-import
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Removed email todo item

Browse Source
This commit is contained in:
Frigyes06 2024-01-11 01:20:37 +00:00
parent b8138baa51
commit 8c4c81e6fa
1 changed files with 251 additions and 251 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

502
README.md
View File

@ -1,252 +1,252 @@
# Contra Costa County (California) Address Import # Contra Costa County (California) Address Import
See [https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import](https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import) See [https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import](https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import)
## Data Source ## Data Source
https://www.contracosta.ca.gov/1818/GIS (https://gis.cccounty.us/Downloads/General%20County%20Data/CCC_Adddress_Points.zip) https://www.contracosta.ca.gov/1818/GIS (https://gis.cccounty.us/Downloads/General%20County%20Data/CCC_Adddress_Points.zip)
*TODO:* establish licensing because https://gis.cccounty.us/Downloads/General%20County%20Data/CCC_GIS_Disclaimer.pdf says "THIS DATA CONTAINS COPYRIGHTED INFORMATION OF THE COUNTY OF CONTRA COSTA" Already emailed County about licensing because https://gis.cccounty.us/Downloads/General%20County%20Data/CCC_GIS_Disclaimer.pdf says "THIS DATA CONTAINS COPYRIGHTED INFORMATION OF THE COUNTY OF CONTRA COSTA"
*TODO:* see if there is updated data, this was modified 8/27/2019 *TODO:* see if there is updated data, this was modified 8/27/2019
## Field Mapping ## Field Mapping
`street_num` -> `addr:housenumber` `street_num` -> `addr:housenumber`
`trim(array_to_string(array(prefix_typ,prefix_dir,street_nam,street_typ,suffix_dir),' '))` `trim(array_to_string(array(prefix_typ,prefix_dir,street_nam,street_typ,suffix_dir),' '))`
-> ->
`addr:street` `addr:street`
`unit_numbe` -> `addr:unit` `unit_numbe` -> `addr:unit`
`city` -> `addr:city` `city` -> `addr:city`
`zip_code` -> `addr:postcode` `zip_code` -> `addr:postcode`
## Instructions ## Instructions
* Download QGIS and JOSM. Download the MapWithAI plugin in JSOM and set your JOSM preferences to validate everything always, especially Addresses on upload and on demand, and at the bottom Mismatched street/street address (MapWithAI) on upload and on demand. * Download QGIS and JOSM. Download the MapWithAI plugin in JSOM and set your JOSM preferences to validate everything always, especially Addresses on upload and on demand, and at the bottom Mismatched street/street address (MapWithAI) on upload and on demand.
* Open the original data in QGIS * Open the original data in QGIS
* Format OSM fields with QGIS functions to have proper capitalization and full spellings without extraneous whitespace, based on original fields. For example OSM uses names like North Main Street, not N MAIN ST. All fields are of the QGIS type "text" even if they're numbers. * Format OSM fields with QGIS functions to have proper capitalization and full spellings without extraneous whitespace, based on original fields. For example OSM uses names like North Main Street, not N MAIN ST. All fields are of the QGIS type "text" even if they're numbers.
* You can use the Attribute Table's Field Calculator for this; you can copy-paste the QGIS Functions script below into the Function Editor and then use the Expression tab to create new, formatted virtual fields. Don't worry if the field name limit is too short, it can be fixed in JOSM. * You can use the Attribute Table's Field Calculator for this; you can copy-paste the QGIS Functions script below into the Function Editor and then use the Expression tab to create new, formatted virtual fields. Don't worry if the field name limit is too short, it can be fixed in JOSM.
* The Addresses shapefile is recorded in a California-specific CRS, make sure your project is set to WGS84 and reproject upon opening the Shapefile. * The Addresses shapefile is recorded in a California-specific CRS, make sure your project is set to WGS84 and reproject upon opening the Shapefile.
* Export to Geojson **selecting only the OSM-formatted fields we want**. * Export to Geojson **selecting only the OSM-formatted fields we want**.
* Here you can rename any temporary or misnamed columns like `addr:house` to `addr:housenumber` etc. * Here you can rename any temporary or misnamed columns like `addr:house` to `addr:housenumber` etc.
* Ensure the export file is in the `EPSG:4326 - WGS84` CRS. * Ensure the export file is in the `EPSG:4326 - WGS84` CRS.
* Open the GeoJSON in JSOM. It's suggested to begin with roads first, addresses second, so the addresses can be placed in context. * Open the GeoJSON in JSOM. It's suggested to begin with roads first, addresses second, so the addresses can be placed in context.
* Select a small region to work on: one neighborhood or smaller. For this import, we are assuming that only newly-constructed small residential areas will be imported, not main roads or commercial areas or areas with significant existing map data. * Select a small region to work on: one neighborhood or smaller. For this import, we are assuming that only newly-constructed small residential areas will be imported, not main roads or commercial areas or areas with significant existing map data.
* Download the area you're working on from OSM, into a new Data Layer (not your geojson layer.) * Download the area you're working on from OSM, into a new Data Layer (not your geojson layer.)
* Run the validation routine on the geojson data layer to see if there are any duplicate addresses within the data. Address them first. * Run the validation routine on the geojson data layer to see if there are any duplicate addresses within the data. Address them first.
* Ensure the tags are correct and good. (Shapefiles have a character limit and sometimes don't like colons, so double check that `addr:house` is `addr:housenumber`, `addr:postc` is `addr:postcode`, `addr:stree` is `addr:street`, etc.) * Ensure the tags are correct and good. (Shapefiles have a character limit and sometimes don't like colons, so double check that `addr:house` is `addr:housenumber`, `addr:postc` is `addr:postcode`, `addr:stree` is `addr:street`, etc.)
* Mass-add any new tags if desired (like `state=CA`) * Mass-add any new tags if desired (like `state=CA`)
* Remove any spurious tags that may have been brought over in the import (if it's not in the OSM Wiki, we don't want it.) * Remove any spurious tags that may have been brought over in the import (if it's not in the OSM Wiki, we don't want it.)
* Select all features to be imported at this time and leave them selected until the merge step below. * Select all features to be imported at this time and leave them selected until the merge step below.
* Make sure that your selection window only includes nodes, no relations. * Make sure that your selection window only includes nodes, no relations.
* Press ctrl-shift-M to merge into the OSM data layer. There will be a warning, but click OK; we will be extra careful about validating the merge in the next steps. * Press ctrl-shift-M to merge into the OSM data layer. There will be a warning, but click OK; we will be extra careful about validating the merge in the next steps.
* For addresses, remove any address nodes that seem to not reflect reality or be placed far from the street bearing their name: it's better to not have 123 Adams Street mapped at all, than to claim that 123 Adams Street is hovering over someone's newly-built house at 321 Franklin Avenue, 200 feet away from Adams Street. (Cities often won't remove old addresses, leading to confusion when new streets are built.) Future OSMers can always add more/better data, but it's super confusing for everyone to try and fix wrong data. * For addresses, remove any address nodes that seem to not reflect reality or be placed far from the street bearing their name: it's better to not have 123 Adams Street mapped at all, than to claim that 123 Adams Street is hovering over someone's newly-built house at 321 Franklin Avenue, 200 feet away from Adams Street. (Cities often won't remove old addresses, leading to confusion when new streets are built.) Future OSMers can always add more/better data, but it's super confusing for everyone to try and fix wrong data.
* Check the edges of the imported areas to ensure that there aren't any undownloaded areas along the edges of where you're about to import. * Check the edges of the imported areas to ensure that there aren't any undownloaded areas along the edges of where you're about to import.
* Use the JOSM validator to ensure no errors in imported data. Warnings about existing data separate from the import can be ignored. The easiest way to do this is to click the upload button and then cancel the upload (Upload validations only validate changes.) * Use the JOSM validator to ensure no errors in imported data. Warnings about existing data separate from the import can be ignored. The easiest way to do this is to click the upload button and then cancel the upload (Upload validations only validate changes.)
* If there are duplicate house numbers in the data, investigate and remove the more-unlikely node or both nodes. * If there are duplicate house numbers in the data, investigate and remove the more-unlikely node or both nodes.
* Click upload * Click upload
* Make sure there are no erroneous Relations or other unwanted objects about to be uploaded. * Make sure there are no erroneous Relations or other unwanted objects about to be uploaded.
* Use a descriptive changeset message like "Addresses in Contra Costa County #cccimport" * Use a descriptive changeset message like "Addresses in Contra Costa County #cccimport"
* Set the Source to be "Contra Costa County GIS" * Set the Source to be "Contra Costa County GIS"
* You can easily copy-paste the below into the Settings tab: * You can easily copy-paste the below into the Settings tab:
``` ```
comment=Addresses in Contra Costa County #cccimport comment=Addresses in Contra Costa County #cccimport
import=yes import=yes
website=https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import website=https://wiki.openstreetmap.org/wiki/Contra_Costa_County_Address_Import
source=Contra Costa County GIS source=Contra Costa County GIS
source:url=https://git.zyphon.com/public/contra-costa-import source:url=https://git.zyphon.com/public/contra-costa-import
``` ```
* Review imported data in Achavi or Osmcha to ensure it looks proper. * Review imported data in Achavi or Osmcha to ensure it looks proper.
## QGIS Script ## QGIS Script
``` ```
import qgis.core import qgis.core
import qgis.gui import qgis.gui
import re import re
# #
# This will keep street names like SR 574A as SR 574A however # This will keep street names like SR 574A as SR 574A however
# will lowercase other number-digit suffixes with <2 or >4 numbers # will lowercase other number-digit suffixes with <2 or >4 numbers
# or >1 suffix-letters, like 12th Street or 243rd Ave. # or >1 suffix-letters, like 12th Street or 243rd Ave.
# #
@qgsfunction(args='auto', group='Custom', referenced_columns=[]) @qgsfunction(args='auto', group='Custom', referenced_columns=[])
def getstreetfromaddress(value1, feature, parent): def getstreetfromaddress(value1, feature, parent):
parts = value1.split() parts = value1.split()
parts.pop(0) # Ignore the first bit (i.e. "123" in "123 N MAIN ST") parts.pop(0) # Ignore the first bit (i.e. "123" in "123 N MAIN ST")
parts = map(formatstreetname, parts) parts = map(formatstreetname, parts)
return " ".join(parts) return " ".join(parts)
@qgsfunction(args='auto', group='Custom', referenced_columns=[]) @qgsfunction(args='auto', group='Custom', referenced_columns=[])
def formatstreet(value1, feature, parent): def formatstreet(value1, feature, parent):
parts = value1.split() parts = value1.split()
# Handle the special case of a street name starting with "ST" # Handle the special case of a street name starting with "ST"
# which is almost always "Saint __" and not "Street __" # which is almost always "Saint __" and not "Street __"
if parts[0].upper() == "ST": if parts[0].upper() == "ST":
parts[0] = "Saint" parts[0] = "Saint"
parts = map(formatstreetname, parts) parts = map(formatstreetname, parts)
return " ".join(parts) return " ".join(parts)
# Internal function # Internal function
def formatstreetname(name): def formatstreetname(name):
nameUp = name.upper() nameUp = name.upper()
# Acronyms # Acronyms
if nameUp == "CR": if nameUp == "CR":
return "County Road" return "County Road"
if nameUp == "SR": if nameUp == "SR":
return "SR" # State Route return "SR" # State Route
if nameUp == "NFS": if nameUp == "NFS":
return "NFS" # National Forest Service? return "NFS" # National Forest Service?
if nameUp == "US": if nameUp == "US":
return "US" return "US"
# Directions # Directions
if nameUp == "N": if nameUp == "N":
return "North" return "North"
if nameUp == "NE": if nameUp == "NE":
return "Northeast" return "Northeast"
if nameUp == "E": if nameUp == "E":
return "East" return "East"
if nameUp == "SE": if nameUp == "SE":
return "Southeast" return "Southeast"
if nameUp == "S": if nameUp == "S":
return "South" return "South"
if nameUp == "SW": if nameUp == "SW":
return "Southwest" return "Southwest"
if nameUp == "W": if nameUp == "W":
return "West" return "West"
if nameUp == "NW": if nameUp == "NW":
return "Northwest" return "Northwest"
# Names # Names
if nameUp == "MACLEAY": if nameUp == "MACLEAY":
return "MacLeay" return "MacLeay"
if nameUp == "MCCLAINE": if nameUp == "MCCLAINE":
return "McClaine" return "McClaine"
if nameUp == "MCAHREN": if nameUp == "MCAHREN":
return "McAhren" return "McAhren"
if nameUp == "MCCAMMON": if nameUp == "MCCAMMON":
return "McCammon" return "McCammon"
if nameUp == "MCCLELLAN": if nameUp == "MCCLELLAN":
return "McClellan" return "McClellan"
if nameUp == "MCCOY": if nameUp == "MCCOY":
return "McCoy" return "McCoy"
if nameUp == "MCDONALD": if nameUp == "MCDONALD":
return "McDonald" return "McDonald"
if nameUp == "MCGEE": if nameUp == "MCGEE":
return "McGee" return "McGee"
if nameUp == "MCGILCHRIST": if nameUp == "MCGILCHRIST":
return "McGilchrist" return "McGilchrist"
if nameUp == "MCINTOSH": if nameUp == "MCINTOSH":
return "McIntosh" return "McIntosh"
if nameUp == "MCKAY": if nameUp == "MCKAY":
return "McKay" return "McKay"
if nameUp == "MCKEE": if nameUp == "MCKEE":
return "McKee" return "McKee"
if nameUp == "MCKENZIE": if nameUp == "MCKENZIE":
return "McKenzie" return "McKenzie"
if nameUp == "MCKILLOP": if nameUp == "MCKILLOP":
return "McKillop" return "McKillop"
if nameUp == "MCKINLEY": if nameUp == "MCKINLEY":
return "McKinley" return "McKinley"
if nameUp == "MCKNIGHT": if nameUp == "MCKNIGHT":
return "McKnight" return "McKnight"
if nameUp == "MCLAUGHLIN": if nameUp == "MCLAUGHLIN":
return "McLaughlin" return "McLaughlin"
if nameUp == "MCLEOD": if nameUp == "MCLEOD":
return "McLeod" return "McLeod"
if nameUp == "MCMASTER": if nameUp == "MCMASTER":
return "McMaster" return "McMaster"
if nameUp == "MCNARY": if nameUp == "MCNARY":
return "McNary" return "McNary"
if nameUp == "MCNAUGHT": if nameUp == "MCNAUGHT":
return "McNaught" return "McNaught"
if nameUp == "O'BRIEN": if nameUp == "O'BRIEN":
return "O'Brien" return "O'Brien"
if nameUp == "O'CONNOR": if nameUp == "O'CONNOR":
return "O'Connor" return "O'Connor"
if nameUp == "O'NEIL": if nameUp == "O'NEIL":
return "O'Neil" return "O'Neil"
if nameUp == "O'TOOLE": if nameUp == "O'TOOLE":
return "O'Toole" return "O'Toole"
# Suffixes # Suffixes
if nameUp == "AV": if nameUp == "AV":
return "Avenue" return "Avenue"
if nameUp == "AVE": if nameUp == "AVE":
return "Avenue" return "Avenue"
if nameUp == "BLVD": if nameUp == "BLVD":
return "Boulevard" return "Boulevard"
if nameUp == "BV": if nameUp == "BV":
return "Boulevard" return "Boulevard"
if nameUp == "BND": if nameUp == "BND":
return "Bend" return "Bend"
if nameUp == "CIR": if nameUp == "CIR":
return "Circle" return "Circle"
if nameUp == "CR": if nameUp == "CR":
return "Circle" return "Circle"
if nameUp == "CT": if nameUp == "CT":
return "Court" return "Court"
if nameUp == "DR": if nameUp == "DR":
return "Drive" return "Drive"
if nameUp == "FLDS": if nameUp == "FLDS":
return "Fields" return "Fields"
if nameUp == "GRV": if nameUp == "GRV":
return "Grove" return "Grove"
if nameUp == "HL": if nameUp == "HL":
return "Hill" return "Hill"
if nameUp == "HOLW": if nameUp == "HOLW":
return "Hollow" return "Hollow"
if nameUp == "HW": if nameUp == "HW":
return "Highway" return "Highway"
if nameUp == "HWY": if nameUp == "HWY":
return "Highway" return "Highway"
if nameUp == "HY": if nameUp == "HY":
return "Highway" return "Highway"
if nameUp == "LN": if nameUp == "LN":
return "Lane" return "Lane"
if nameUp == "LOOP": if nameUp == "LOOP":
return "Loop" return "Loop"
if nameUp == "LP": if nameUp == "LP":
return "Loop" return "Loop"
if nameUp == "MT": if nameUp == "MT":
return "Mount" return "Mount"
if nameUp == "MTN": if nameUp == "MTN":
return "Mountain" return "Mountain"
if nameUp == "PATH": if nameUp == "PATH":
return "Path" return "Path"
if nameUp == "PL": if nameUp == "PL":
return "Place" return "Place"
if nameUp == "RD": if nameUp == "RD":
return "Road" return "Road"
if nameUp == "RUN": if nameUp == "RUN":
return "Run" return "Run"
if nameUp == "SQ": if nameUp == "SQ":
return "Square" return "Square"
if nameUp == "ST": if nameUp == "ST":
return "Street" return "Street"
if nameUp == "TER": if nameUp == "TER":
return "Terrace" return "Terrace"
if nameUp == "TR": if nameUp == "TR":
return "Trail" return "Trail"
if nameUp == "TRL": if nameUp == "TRL":
return "Trail" return "Trail"
if nameUp == "VW": if nameUp == "VW":
return "View" return "View"
if nameUp == "WAY": if nameUp == "WAY":
return "Way" return "Way"
if nameUp == "WY": if nameUp == "WY":
return "Way" return "Way"
if nameUp == "XING": if nameUp == "XING":
return "Crossing" return "Crossing"
if re.match('^[0-9]{2,4}[A-Za-z]$', name) != None: if re.match('^[0-9]{2,4}[A-Za-z]$', name) != None:
return name return name
return name #.capitalize() return name #.capitalize()
``` ```