Peering with Verizon (in Los Angeles)

[snarked]

Any thoughts on peering with Verizon (any of their AS's, but 19262 preferred) anywhere, let alone Los Angeles?  Alternatively, adjusting an existing peer (Telia) to also peer in LA?

From Verizon to HE, the AS map (courtesy of robtex.com) indicates that you share two AS's in common:  GBLX (3549, Global Crossing) and TELIANET (1299).

Packets having endpoints in Los Angeles go to San Jose and back!

  Verizon DSL <=> alter.net <=> telia.net <=> HE <=> xfernet.net (an HE transit customer)

The same thing happens if I were to trace to HE's Los Angeles tunnel server (66.220.18.42).  Looks like a tunnel from up north (SF Bay area) would be better for my home location!

Traceroute:
 1  My internal router at home  0.396 ms  0.303 ms  0.267 ms
 2  pool-71-160-18-32.lsanca.dsl-w.verizon.net (71.160.18.32)  1.076 ms  1.072 ms  0.891 ms
 3  l100.lsanca-dsl-15.verizon-gni.net (71.106.208.1)  40.139 ms  18.536 ms  19.719 ms
 4  g3-2-2215.lsanca-lcr-12.verizon-gni.net (130.81.46.44)  20.938 ms  20.583 ms  19.944 ms
 5  so-4-3-0-0.lax01-bb-rtr2.verizon-gni.net (130.81.29.246)  20.458 ms  21.638 ms  21.109 ms
 6  0.so-6-2-0.xt2.lax7.alter.net (152.63.112.49)  21.172 ms  21.985 ms  21.657 ms
 7  0.so-5-2-0.xl4.sjc7.alter.net (152.63.48.2)  33.264 ms  33.905 ms  33.494 ms
 8  pos5-0-0.gw4.sjc7.alter.net (152.63.49.173)  33.784 ms pos7-0-0.gw4.sjc7.alter.net (152.63.48.245)  31.927 ms  29.967 ms
 9  teliasonera-test-gw.alter.net (157.130.215.70)  30.072 ms  35.368 ms  30.304 ms
10  hurricane-113209-sjo-bb1.c.telia.net (213.248.86.54)  32.265 ms  32.685 ms  39.379 ms
11  10gigabitethernet1-1.core1.lax1.he.net (72.52.92.22)  44.591 ms  44.361 ms  44.655 ms
12  t0-1.cs1.lax1.xfernet.net (65.19.143.10)  41.213 ms  41.539 ms  42.631 ms
13  g50.ar9.lax1.xfernet.net (67.43.160.90)  44.589 ms  45.862 ms  44.387 ms
14  www.snarked.org (67.43.172.253)  43.809 ms  45.376 ms  43.559 ms

Other Direction:
From www.snarked.org [67.43.172.252] to 71.160.18.32
 1  snarked.ipv4-gw.xfernet.net (67.43.172.249)  1.265 ms  0.551 ms  0.386 ms
 2  vl50.cs2.lax1.xfernet.net (67.43.160.97)  0.344 ms  0.892 ms  0.275 ms
 3  g2-16.cs1.lax1.xfernet.net (67.43.160.205)  0.996 ms  0.435 ms  0.262 ms
 4  10gigabitethernet2-2.core1.lax1.he.net (65.19.143.9)  0.407 ms  9.488 ms  0.365 ms
 5  las-bb1-link.telia.net (213.248.70.37)  10.840 ms  10.845 ms  16.444 ms
 6  sjo-bb1-link.telia.net (213.248.80.16)  10.831 ms  10.905 ms  10.752 ms
 7  GigabitEthernet2-0-0.GW4.SJC7.ALTER.NET (157.130.215.69)  10.821 ms  10.799 ms  10.768 ms
 8  0.so-0-1-0.XL3.SJC7.ALTER.NET (152.63.49.170)  10.892 ms  10.876 ms  10.844 ms
 9  0.so-6-0-0.SJC01-BB-RTR1.VERIZON-GNI.NET (152.63.49.150)  11.331 ms 0.so-3-3-0.SJC01-BB-RTR1.verizon-gni.net (152.63.1.194)  12.715 ms 0.so-6-0-0.SJC01-BB-RTR1.VERIZON-GNI.NET (152.63.49.150)  11.318 ms
10  ge-2-2-0-0.LAX01-BB-RTR1.verizon-gni.net (130.81.17.37)  23.706 ms  22.489 ms  23.521 ms
11  P12-0.LSANCA-LCR-03.verizon-gni.net (130.81.28.226)  23.932 ms  22.362 ms  23.911 ms
12  P13-0.LSANCA-LCR-05.verizon-gni.net (130.81.30.210)  21.427 ms  21.565 ms  22.991 ms
13  P14-0.LSANCA-LCR-07.verizon-gni.net (130.81.27.231)  21.596 ms  21.479 ms  22.950 ms
14  P14-0.LSANCA-LCR-09.verizon-gni.net (130.81.27.165)  146.101 ms  117.537 ms  207.258 ms
15  P12-0-0.LSANCA-LCR-11.verizon-gni.net (130.81.27.191)  23.927 ms  22.889 ms  22.480 ms
16  G12-0-2115.LSANCA-DSL-15.verizon-gni.net (130.81.46.47)  23.550 ms  23.716 ms  23.481 ms
17  pool-71-160-18-32.lsanca.dsl-w.verizon.net (71.160.18.32)  41.220 ms  41.560 ms  40.618 ms

I note that you are at LAIIX and "One Wilshire" but Verizon, alter.net, and telia.net are NOT.  None of you are at CIIX.  However, as Verizon is also a local telco in the area, can they be invited?

[jimb]

Heh.  Probably would be better for you to use one of the Fremont tunnels.  I'll wave to your packets as they go by my house.   

Do you have FiOS btw?  I know the dname says verizon-dsl.  A friend down in LA just got it.  Sooo nice.  Except they don't offer statics.

[brad]

Heh.  Probably would be better for you to use one of the Fremont tunnels.  I'll wave to your packets as they go by my house.   

Do you have FiOS btw?  I know the dname says verizon-dsl.  A friend down in LA just got it.  Sooo nice.  Except they don't offer statics.

Strange. I know someone in Tampa, FL with FiOS and he definitely has statics. Its a requirement for what the service is being used for.

[jimb]

Heh.  Probably would be better for you to use one of the Fremont tunnels.  I'll wave to your packets as they go by my house.   

Do you have FiOS btw?  I know the dname says verizon-dsl.  A friend down in LA just got it.  Sooo nice.  Except they don't offer statics.

Strange. I know someone in Tampa, FL with FiOS and he definitely has statics. Its a requirement for what the service is being used for.

Maybe it's business class service?  I know there's "FiOS Business" which offers statics.  A friend of mine has residential and he told me they weren't an option.  Perhaps he was mistaken?

[snarked]

1)  No, it's not FIOS.  It's DSL.  DSL only became available in my neighborhood in 2006.  We'll probably be the last to see FIOS in LA City.

2)  The LA tunnel is the closest for my co-located server.  HE is one of 3 transit carriers.  Traceroute:

From www.snarked.org [67.43.172.253] to 66.220.18.42
 1  snarked.ipv4-gw.xfernet.net (67.43.172.249)  1.462 ms  0.554 ms  1.048 ms (co-lo facility router)
 2  vl50.cs2.lax1.xfernet.net (67.43.160.97)  0.511 ms  0.422 ms  0.258 ms       (co-lo facility router)
 3  g2-16.cs1.lax1.xfernet.net (67.43.160.205)  0.893 ms  0.429 ms  0.266 ms   (co-lo facility router)
 4  10gigabitethernet2-2.core1.lax1.he.net (65.19.143.9)  0.378 ms  0.412 ms  0.291 ms
 5  tserv15.lax1.ipv6.he.net (66.220.18.42)  0.932 ms  1.901 ms  0.348 ms        (3 miles away)

3)  Although these are really tunnelbroker forums, the issue I have is also from home to my co-located server going via San Jose when both endpoints are in Los Angeles.  As HE is also the transit carrier at one end, I thought they might want to do something about it.  If they were to peer with Verizon (or alter.net) in LA, then local transit would be faster, including to/from the LA tunnel server.

[jimb]

Ah right I misunderstood your message.  I thought you were saying that path from your home DSL to your local HE TS was similar to the traceroutes you were showing due to some BGP issue or peering arrangement or whatever.

Seems like Xfernet isn't very well connected eh?  They could also do a bit more peering and make the situation better for you it seems.

[snarked]

NO, you got it right.  The path to/from home is highly similar when comparing HE's Los Angeles TS and my co-located box.  Both go to northern California and back.

It's not xfernet's connectivity in question.  It's the path(s) between HE and Verizon that is.
(Xfernet's other peering partners include Cogent and Mzima, which are well connected).
(Verizon itself is well connected, but not to HE.)

[brad]

Any thoughts on peering with Verizon (any of their AS's, but 19262 preferred) anywhere, let alone Los Angeles?  Alternatively, adjusting an existing peer (Telia) to also peer in LA?

Considering the fact that HE has no peering with any of the Tier 1 providers the chances of that happening at the moment are like zero.

Also peering with VZB's overlay network AS for v6 would be good in general but woudn't help your v4 path to HE's POP.

Which makes me wonder why HE relies so heavily on their transit providers. Has HE even made any attempts to try to establish peering with any of the Tier 1 providers?

[mleber]

Re: Peering with Verizon (in Los Angeles)

[brad]

That isn't accurate.  There is the IPv6 Internet and the IPv4 Internet.

In the IPv6 world, Hurricane got started very early (2000), replaced our whole network to support wirespeed IPv6 in the core in 2007, and now is the largest IPv6 network as measured by number of IPv6 networks we connect (verify here: http://bgp.potaroo.net/v6/as6447/bgp-as-adj.txt ) as well as measured by number of IPv6 customer prefixes announced.

In the IPv6 world we peer with all of the top IPv6 players and many networks considered tier1 in IPv4.  IPv6 is a bit of an "global Internet routing table reset" because many former tier1 IPv4 networks haven't even started deploying IPv6 and of those that have many are still running limited tunnel networks without complete routing tables, instead of running dual stack in their core and aggressively peering enough to get a full IPv6 routing table.  Some otherwise global networks only IPv6 peer at one or just a few locations in the world, even if their network overlaps Hurricane in 20 locations.  To the extent they do peer, you can be reassured that they do care about IPv6 routing and want to ensure it works.  As they start to deploy IPv6 if you are their customer you can help improve the experience by letting them know in writing where they can improve their routing.  If you are an IP Transit customer of a legacy tier1 network, one of your challenges will be getting them to bifurcate their peering policy between the IPv4 Internet were they are dominant, and IPv6 were they may be barely extant (depending on who they are).

In the IPv4 world, Hurricane has 4000+ BGP sessions with 1100+ networks at 24+ exchange points in North America, Europe, and Asia.  However, that said, the core of the IPv4 Internet is a bit of an oligopoly with the core players largely determined around 1994 with the existing players being either those networks or the corporations that acquired those networks.  Consider this, even China Telecom with more Internet users than the US (and more mobile device users than the entire US population) is not a tier1 transit network in IPv4.

In this light, that fact that Hurricane even usually ranks (currently #8, it fluctuates week to week) in the top 10 IPv4 networks in the world according to CAIDA (verify here: http://as-rank.caida.org/ ) is a veritable miracle.  We've clawed our way up this far AND we haven't stopped growing in IPv4 (we are bringing online Stockholm and Zurich this week, we brought Tokyo online in the last few weeks), even if you ignore all the excitement happening with IPv6. 

To answer more directly, it is Hurricane's mission to provide the best Internet we can to as many people as possible.  In light of that, we read and extensively discuss internally comments to this board and feedback given to us in tickets about routing.  These comments have caused us to add circuits to shorten latency (we now have a New York to Los Angeles that doesn't show up on our network map), add tunnel servers, replace and improve the tunnel server platform, pursue peering with specific networks more aggressively, and add more peering with existing peers.

I admit I wasn't as clear as I could be. I was referring to v4 only. v6 is a completely different ball game and HE being at the top of the food chain so to speak changes things in their favor big time going forward. I also am aware of the extensive peering with various other networks.. just not the Tier 1 carriers (again, for v4). HE growing in size both in the US and abroad helps to increase the networks size, coverage and helps with added leverage when negotiating additional peering sessions. I am aware of what HE has accomplished and I'm not trying to downplay that in any way. Just curious what the situation is with HE peering with the Tier 1 carriers.

Although it is difficult to get into the official Tier 1 space it isn't impossible. SAVVIS, NTT and TeliaSonera are examples of networks that were not Tier 1 status until not that long ago in the bigger picture and either grew on their own (Telia) or purchased assets of another company to get where they are (NTT - Verio) (SAVVIS - former C&W USA -> MCI AS) now. Cogent, XO, AboveNet and a few others are pretty close to being considered in the same space though not quite there yet.

[snarked]

Now if only those tier 1 carriers would join the exchange points, ....

What I suggested was that Verizon or alter.net join one of the two peering points in LA that HE participates with, and I would see a decrease of hops plus the traffic would stay local.  As Los Angeles is the second largest (by population) area in the U.S., proper integration of the infrastructure should produce routes that STAY in the LA area when the endpoints are both in LA.  When that doesn't happen, something is "wrong."  ;-)

[kriteknetworks]

Heh, its only wrong if their isn't mutual benefit, aka money. Routing efficiency is dependant on money changing hands, not packet flows

[brad]

Now if only those tier 1 carriers would join the exchange points, ....

What I suggested was that Verizon or alter.net join one of the two peering points in LA that HE participates with, and I would see a decrease of hops plus the traffic would stay local.  As Los Angeles is the second largest (by population) area in the U.S., proper integration of the infrastructure should produce routes that STAY in the LA area when the endpoints are both in LA.  When that doesn't happen, something is "wrong."  ;-)

Depending on the Tier 1 network they don't necessarily do much if any public peering at all. Their selective / restrictive peering policies stipulate peering at typically OC48 or OC192 speeds minimum depending whether it is in US or Europe. Because of such high speeds that implies only private peering. Also other things like minimum traffic levels, connections at multiple locations, over multiple continents where and when the networks do exist, required traffic ratios, etc. But even if you do meet the stipulated requirements there is no guarantee they will peer with you. The other option which some networks go for when they're insistent on having peering anyway and don't meet the requirement such as traffic rations or minimum traffic levels is to have paid peering. But I think the second option essentially will not happen with HE when they already have two good transit providers with interconnection with these other networks.

I find one hop name a little funny "teliasonera-test-gw". Is this one of Telia's settlement free peering links after switching from paid peering to settlement free with VZB? hrmm.

I see VZB is at Equinix LA and One Wilshire and so is TeliaSonera. I can't find a very detailed map for Global Crossing but I can pretty much guarantee they are at those two locations as well since they're very common places to interconnection with other networks and customers.

I see Global Crossing peers with VZB in LA and from Global Crossing to VZB traffic goes via LA without going back to SJC.

[snarked]

Note that I determined peering at the exchanges by looking at the exchange points' web sites list of participants.  Verizon wasn't listed at any of them.

[brad]

Note that I determined peering at the exchanges by looking at the exchange points' web sites list of participants.  Verizon wasn't listed at any of them.

Of course they won't be listed since they don't do pretty much any public peering.