Lumen help

Log Streaming and Management

For most reporting requirements, Lumen Media Portal delivers information in a convenient, easy-to-use tool. For those advanced requirements that need information only available in the logs, Lumen can provide access logs generated by the CDN edge servers. By analyzing access logs, you can view utilization and performance of your websites, applications, content, and hardware/software resources. They can also help you analyze or debug new or errant functionality. Response times, resource usage, traffic, trending, geographic patterns, errors and a wealth of available information can be derived from these access logs. The volume of log data generated by high request-rate properties can be extremely large, so you'll want to plan carefully and consider the resources required to retrieve and analyze the data.

Log Delivery Formats

Lumen provides two methods for delivery of log information. Typically, logs are delivered from the edge to you within five minutes.

Log Streaming: the log streaming processor parses the logs, maps to specified customer endpoint(s), and sends the appropriate messages to the designated customer endpoint(s). Supported customer designated endpoints include (1) a generic, customer-specified host or (2) a 3rd party cloud log analytic solution. The Log Streaming application sends CDN access logs through either an HTTP or HTTPS protocol (configurable as part of activation) in JSON format.

Message Format: JSON

Either JSON messages separated by newline characters or a JSON formatted array messages are used for Log Streaming.

A simple JSON example is as follows (new lines and beginning of line spacing added for clarity, not in final message):

                              {
    "cached":1,
    "cs(Cookie)":"-",
    "cs(Referer)":"http://www.test.com/some/referrer/path",
    "cs(User-Agent)":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5)
     AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36",
    "cs-ip":"127.75.23.245",
    "cs-method":"get",
    "cs-uri":"/test/test.gif?auth=ouAoARQAFoKCAAQABgAIAAoAGAAaR0b24xLnNq",
    "cs-version":"HTTP/1.1",
    "date":"2017-07-12",
    "sc(Content-Type)":"-",
    "sc-bytes":383,
    "status":200,
    "time":"22:27:10",
    "time-taken":0.01,
    "x-disid":2124407,
    "x-headersize":340,
    "cs(Range)":"bytes=0-123456”,
    "x-tcpinfo_rcv_space": 28960,
    "x-tcpinfo_rtt": 14000,
    "x-tcpinfo_rttvar": 7000,
    "x-tcpinfo_snd_cwnd": 32,
    "x-tcwait":245,
    "x-tdwait":0.003
  }
            
        

There is no set number of messages in a batch sent for each POST. Messages will be sent when either the Max Bytes/Message (in bytes) or when the Max Post Interval (in seconds) has been reached since the last HTTP POST. If there are no messages to be sent in a batch, then the system will skip sending on that interval and wait until there is at least one message in the batch. This is done for each process, currently one per machine. Both configuration parameters have defaults defined by Lumen for optimal performance.

The log streaming JSON encoding escapes common HTML characters by default. The following characters are mapped to provided escaped Unicode:

‘&’ : ‘\u0026’
‘<’ : ‘\u003c’
‘>’ : ‘\u003e’
U+2028 (Unicode line separator) : ‘\u2028’
U+2029 (Unicode paragraph separator) : ‘\u2029’

These can commonly appear inside URL query strings or HTML parts sent through the log streaming system:

-- “/example?key=value&hello=world” becomes “/example?key=value\u0026hello=world”

-- “<b>hello world!<\b>” becomes “\u003cb\u003ehello world! \u003c\b\u003e”

It should be noted that customers with large traffic volumes can create a significant volume of logs and must provide a log endpoint capable of ingesting at a high rate. Failure to provide such an endpoint may result in logs being dropped as the CTL log processing system only queues logs for a short period of time if the endpoint is unavailable or unable to keep up.

Log File Delivery: the log file delivery processor parses the logs, maps to specified customers, and sends the appropriate files to the designated customer log file folder on the Lumen Origin Storage Platform (OSP). Log files follow W3C extended log file format. Once the service is enabled for you, the log data is written to a your Origin Storage Platform (OSP) account using your primary alias (e.g. lumen.example.com) (by default, but can be configured) and will be retained until you delete the log files.

the log file format follows the W3C Extended Log File Format defined by the World Wide Web Consortium. By default, messages will be available once the MaxBytesPerFile threshold is met (default set to 1 Gig) or when the MaxWaitSecForFinalWrite is met (default to 15-minute duration) (whichever occurs first).

A simple W3C log example is as follows:

                            #Version: 1.0
#Software: 4.0
#Fields: date time cs-ip cs-method cs-uri status sc-bytes time-taken cs(Referer)
cs(User-Agent) cs(Cookie) x-Custom x-LogGroup x-disid x-extstatus x-headersize cache
cs(Range) x-tcwait x-tcpinfo_rtt x-tcpinfo_rttvar x-tcpinfo_snd_cwnd x-tcpinfo_rcv_space
x-tdwait sc(Content-Type) cs-version
2017-07-23 12:10:02.319 10.10.10.10 GET
http://www.example.com/index.html 200 25574 0.003 “-”
“user-agent/1.1” “-” “-” y 153001 0 803 MISS
“-” 0 29058 3486 162 28960 3 “text/html” “HTTP/1.1”
            
        

Note: the use of the #Software: 4.0 log line. This indicates what version of logging is enabled for the customer. Lumen currently offers multiple W3C log formats – V1, V3 and V4 (default).

Log Delivery

Once MLP completes the log processing operation, logs are delivered to the customer’s respective OSP (osp_origin) account.

The logs are placed in the customer’s OSP account under the /logs directory. Subdirectories are created that follow a date/primary alias naming convention of:
/logs/caching/<year>/<month>/<day>/<primary alias>

Example:
/logs/caching/2015/01/01/level3.example.com

Note: Only primary aliases are used for naming purposes (directory, file name). Data from other aliases will be included in the same directory/file.

The log file uploaded to the customer’s OSP account follows a primary alias/date naming convention of:<primary alias>_<year><month><day><hour><minute><second><CosID>.log[.gz]

Note: The log file name represents the time the file was created by the MLP, it does not represent the time the log entries were created. Typically, the log timestamps will be several minutes earlier than the timestamp in the file name.

Log files are compressed by MLP using GNU Zipped Archive (.gz) compression when sending to a customer’s OSP account, and log files can be retrieved in the customer’s OSP account and managed through a variety of means. For further information, please see Origin Storage Service Information.

Please note that OSP Storage consumed by log data will be billed as part of the customer’s OSP usage at their contracted storage usage rate. The customer is responsible for purging log data from OSP when it is no longer required, otherwise they will continue to be charged for all logs’ data stored, on a monthly basis.

Lumen can also deliver CDN Log Files to our customer’s Amazon Simple Storage Services (Amazon S3) endpoint. Click here to learn more.

Default Log Fields

The following fields are included in the default configuration for log delivery:

Field	Type	Example format	Description
date	<date>	2017-07-09	Date transaction completed
time	<time>	23:27:35:142	Time transaction completed, HH is the hour in 24-hour format
cs-ip	<address>	IPv4: http://xxx.xxx.xxx.xxx IPv6: xxxx:xxxx:xxxx:xxxx: xxxx:xxxx:xxxx:xxxx	IPv4 or IPv6 address of the requesting client, in dot notation
cs-method	<string>	get	Method (e.g., get, head) in client request
cs-uri	<string>	/test	Path fragment of URI
status	<integer>	200	HTTP status code returned to the client, expressed as a single integer Note: legacy name of this field was 'sc status'.
sc-bytes	<integer>	1953	Number of bytes returned in response to request (including headers), expressed as a single integer
time-taken	<float>	0.106	Time taken for transaction to complete (in seconds)
cs(Referer)	<string>	-	Contents of referrer request header
cs(User-Agent)	<string>	urlgrabber/3.1.0 yum/3.2.22	Contents of user agent request header
cs(Cookie)	<string>	-	Content of all cookie headers presented
x-disid	<integer>	2037223	ID of distributor delivering request
x-headersize	<integer>	301	Size of response header in bytes
cached	<integer>	0\|1	Indicates whether the object was served from cache or filled. The status will be either 0 (miss) or 1 (hit) Note: legacy name of this field was 'cache'
cs(Range)	<string>	"bytes=0-123456"	Contents of the range header Note: legacy name of the field was 'x-range'
x-tcwait	<integer>	567	Time (ms) spent waiting for the client to be ready—i.e. data was available to be sent, but the socket to the client was full
x-tcpinfo_rtt	<integer>	678	Smoothed RTT in usecs
x-tcpinfo_rttvar	<integer>	789	RTT variance in usecs
x-tcpinfo_snd_cwnd	<integer>	890	Send congestion window
x-tcpinfo_rcv_space	<integer>	901	Advertised recv window
x-tdwait	<float>	0.222	Time (ms) spent waiting for the upstream to send data
sc(Content Type)	<string>	"what did I just get"	Contents of the content-type header
cs-version	<string>	"HTTP/1.0"	The HTTP version (e.g., HTTP/1.1, h2)

Optional Log Fields

The following fields are optional. They're not included in the default configuration, but you can request that Lumen include them in your configuration:

Field	Type	Example format	Description
cs-scheme	<string>	"http"	The scheme that the client request was made for, commonly http or https.
cs-host	<string>	"http://hello.world.com"	The host of the client's request.
c-asn	<integer>	432	The client ASN as determined from a lookup of the client IP in an online GeoIP database. If unable to determine the value, the system will return zero.
c-city	<string>	"Denver"	The client city as determined from a lookup of the client IP in an online GeoIP database. The city name is provided in all lower case. If unable to determine the value, the system will return "unknown".
c-state	<string>	"CO"	The client state as determined from a lookup of the client IP in an online GeoIP database. The state is provided in an all lower-case full state name and is used for other regional identifies outside of the United States for example, provinces and territories. If unable to determine the value, the system will return "unknown".
c-cc	<string>	"US"	The client country code as determined from a lookup of the client IP in an online GeoIP database. This field follows the ISO 3166-1 alpha-2 standard for two letter country codes. If unable to determine the value, the system will return an empty string.
x-extras	<JSON key value>	"freeform-key":	A freeform set of key value pairs of extra information that can be custom defined. All additional keys must be unique. Note: This field can ONLY be enabled for JSON format.

Deprecated log fields

The following fields have been deprecated. Lumen will only provide these to you if you were previously configured to receive these fields:

Field	Type	Example format	Description
x-Custom	<string>	-	All logs will contain the default value of -.
x-LogGroup	<string>	-	All logs will contain the default value of -.
x-extstatus	<integer>	0	All logs will contain the default value of 0.

Activating CDN Log Collection and Delivery

CDN Log Streaming is a billable service and can be activated by the Lumen Service Activations team once we receive your signed order. Log collection and delivery can be enabled for one or more (or all) primary aliases under a given account as determined by you upon initial configuration.

Limitations

If your peak volume is >50,000 requests/second, you need special approval before activating Log Streaming so we can scale the solution to meet your needs. Contact your Lumen representative and they will work with our internal teams to review your needs.

Log Streaming delivery format is directly tied to the JSON message format. File Delivery format is directly tied to the W3C message format.

Learn more about
Media portal

Explore Media portal

Portal overview